Monitor Docker Environment using InfluxDB, Grafana & Telegraf

0
491
Monitor Docker Environment

Let’s make an interesting monitoring environment for DevOps/Docker environment with Time series metrics. Here we gonna use grafana for dashboard and for database let’s use influxDB and collecting metrics we have lot of tools like collectd, graphite etc. in that let’s pick Telegraf (Owned by InfluxData)

Installing Grafana

Create a persistent storage volume. This ensures that when you destroy and recreate the grafana docker to upgrade it, your configuration will be retained.

# docker run -d -v /var/lib/grafana --name grafana-storage busybox:latest

Create and start the Grafana docker. At this point Grafana will be fully installed and ready to configure.

# docker create \
--name=grafana \
-p 3000:3000 \
--volumes-from grafana-storage \
-e "GF_SECURITY_ADMIN_PASSWORD=<<you-password>>" \
grafana/grafana
# docker start grafana

Docker upgrade script

If you are looking for easy redeployment upgrade script for grafana container refer our repo, devops

Autostart the image – create a systemd startup script

# vi /lib/systemd/system/grafana.service

[Unit]
Description=grafana container
Requires=docker.service
After=docker.service
[Service]
User=foxutech
Restart=on-failure
RestartSec=45
ExecStart=/usr/bin/docker start -a grafana
ExecStop=/usr/bin/docker stop -t 2 grafana
[Install]
WantedBy=multi-user.target

Enable on startup

# systemctl enable grafana.service

If you want to start it up,

# systemctl start grafana

Read More: How to Create InfluxDB and Grafana containers on Linux

Installing InfluxDB

Create your local storage for backup/Storage purpose, to avoid data loss if container got deleted at any case,

# mkdir -p /monitoring/influxdb/conf/
# mkdir -p /monitoring/influxdb/db/

Check your folder ownership (optional)

This may not be required depending on what user you were when you made the folders above and what OS you use. You’ll want this path to be owned by your execution user for the docker servers. Replace user below with your username

# chown user:user -R /docker

Generate the default configuration

You may want to take a look after this after you generate it. There are some useful options to note.

# docker run --rm influxdb influxd config > /monitoring/influxdb/conf/influxdb.conf

Create and start the InfluxDB Container

# docker create \
--name influxdb \
-e PUID=1212 -e PGID=1212 \
-p 8083:8083 -p 8086:8086 \
-v /monitoring/influxdb/conf/influxdb.conf:/etc/influxdb/influxdb.conf:ro \
-v /monitoring/influxdb/db:/var/lib/influxdb \
influxdb -config /etc/influxdb/influxdb.conf
# docker start influxdb

Create Test database

Check out your instance at localhost:8083. You’ll want to create the Test db, to do that just go into the query field and enter:

CREATE DATABASE test

Docker upgrade script

If you are looking for easy redeployment upgrade script for InfluxDB container refer our repo, devops

Autostart the image – create a systemd startup script

# vi /lib/systemd/system/influxdb.service

[Unit]
Description=influxdb container
Requires=docker.service
After=docker.service
[Service]
User=foxutech
Restart=on-failure
RestartSec=45
ExecStart=/usr/bin/docker start -a influxdb
ExecStop=/usr/bin/docker stop -t 2 influxdb
[Install]
WantedBy=multi-user.target

Enable on startup

# systemctl enable influxdb.service

If you want to start it up for testing

# systemctl start influxdb

Docker Statistics

Docker has quickly become part of my daily workflow.  It’s a great way to quickly deploy common services and upgrade frequently used packages while maintaining the integrity of your configuration and volume storage.  I’m not up to nearly 15 docker containers across my single server on two separate VMs.  It’s positively intoxicating to spin up a new piece of software, test it for a bit, and trash it with no trace.

In order to do this, I’m going to walk you through deploying the Telegraf docker image. Telegraf is actually a good replacement for most of the custom scripts you’ve seen me list here. When it comes to SNMP data, you can have Telegraf capture all of this and drop it into a InfluxDB. I haven’t spent a tremendous amount of time with it yet, and admittedly the last thing I setup was my Docker statistics. Knowing what I know now, I’ll likely be making more post consolidating the previously used scripts into the one Telegraf instances to capture all of the SNMP data I’m getting manually. For now, the scripts work very well (and are light on resources).

Install Telegraf with Docker

Create the docker volume storage location

# mkdir -p /monitoring/telegraf/

Generate the default configuration file

# docker run --rm telegraf -sample-config > /monitoring/telegraf/telegraf.conf

Modify this file to fit your needs, lots of modification at the top for your environment. In particular look for these two things.

Output Plugin

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
urls = ["http://localhost:8086"] # required
database = "telegraf" # required
precision = "s"
retention_policy = "default"
write_consistency = "any"timeout = "5s"

Docker Configuration

[[inputs.docker]]
## Docker Endpoint
##   To use TCP, set endpoint = "tcp://[ip]: [port]"
##   To use environment variables (ie, docker-machine), set endpoint = "ENV"
#endpoint = "unix:///var/run/docker.sock"
#you may need to use the actual IP here
endpoint = "tcp://localhost:2375" 
## Only collect metrics for these containers, collect all if empty
container_names = []
## Timeout for docker list, info, and stats commands
timeout = "30s"

Docker upgrade script

If you are looking for easy redeployment upgrade script for Telegraf container refer our repo, devops

Autostart the image – create a systemd startup script

# vi /lib/systemd/system/telegraf.service

[Unit]
Description=telegraf container
Requires=docker.service
After=docker.service
[Service]
User=foxutech
Restart=on-failure
RestartSec=45Exec
Start=/usr/bin/docker start -a telegraf
ExecStop=/usr/bin/docker stop -t 2 telegraf
[Install]
WantedBy=multi-user.target

Enable on startup

# systemctl enable telegraf.service

If you want to start it up for testing

# systemctl start telegraf

Once telegraf starts, you should see data populating in your influxdb.

NO COMMENTS