Let’s make an interesting monitoring environment for DevOps/Docker environment with Time series metrics. Here we gonna use grafana for dashboard and for database let’s use influxDB and collecting metrics we have lot of tools like collectd, graphite etc. in that let’s pick Telegraf (Owned by InfluxData)
Installing Grafana
Create a persistent storage volume. This ensures that when you destroy and recreate the grafana docker to upgrade it, your configuration will be retained.
# docker run -d -v /var/lib/grafana --name grafana-storage busybox:latest
Create and start the Grafana docker. At this point Grafana will be fully installed and ready to configure.
# docker create \
--name=grafana \
-p 3000:3000 \
--volumes-from grafana-storage \
-e "GF_SECURITY_ADMIN_PASSWORD=<<you-password>>" \
grafana/grafana
# docker start grafana
Docker upgrade script
If you are looking for easy redeployment upgrade script for grafana container refer our repo, devops
Autostart the image – create a systemd startup script
# vi /lib/systemd/system/grafana.service
[Unit] Description=grafana container Requires=docker.service After=docker.service [Service] User=foxutech Restart=on-failure RestartSec=45 ExecStart=/usr/bin/docker start -a grafana ExecStop=/usr/bin/docker stop -t 2 grafana [Install] WantedBy=multi-user.target
Enable on startup
# systemctl enable grafana.service
If you want to start it up,
# systemctl start grafana
Read More: How to Create InfluxDB and Grafana containers on Linux
Installing InfluxDB
Create your local storage for backup/Storage purpose, to avoid data loss if container got deleted at any case,
# mkdir -p /monitoring/influxdb/conf/ # mkdir -p /monitoring/influxdb/db/
Check your folder ownership (optional)
This may not be required depending on what user you were when you made the folders above and what OS you use. You’ll want this path to be owned by your execution user for the docker servers. Replace user below with your username
# chown user:user -R /docker
Generate the default configuration
You may want to take a look after this after you generate it. There are some useful options to note.
# docker run --rm influxdb influxd config > /monitoring/influxdb/conf/influxdb.conf
Create and start the InfluxDB Container
# docker create \ --name influxdb \ -e PUID=1212 -e PGID=1212 \ -p 8083:8083 -p 8086:8086 \ -v /monitoring/influxdb/conf/influxdb.conf:/etc/influxdb/influxdb.conf:ro \ -v /monitoring/influxdb/db:/var/lib/influxdb \ influxdb -config /etc/influxdb/influxdb.conf
# docker start influxdb
Create Test database
Check out your instance at localhost:8083. You’ll want to create the Test db, to do that just go into the query field and enter:
CREATE DATABASE test
Docker upgrade script
If you are looking for easy redeployment upgrade script for InfluxDB container refer our repo, devops
Autostart the image – create a systemd startup script
# vi /lib/systemd/system/influxdb.service
[Unit] Description=influxdb container Requires=docker.service After=docker.service [Service] User=foxutech Restart=on-failure RestartSec=45 ExecStart=/usr/bin/docker start -a influxdb ExecStop=/usr/bin/docker stop -t 2 influxdb [Install] WantedBy=multi-user.target
Enable on startup
# systemctl enable influxdb.service
If you want to start it up for testing
# systemctl start influxdb
Docker Statistics
Docker has quickly become part of my daily workflow. It’s a great way to quickly deploy common services and upgrade frequently used packages while maintaining the integrity of your configuration and volume storage. I’m not up to nearly 15 docker containers across my single server on two separate VMs. It’s positively intoxicating to spin up a new piece of software, test it for a bit, and trash it with no trace.
In order to do this, I’m going to walk you through deploying the Telegraf docker image. Telegraf is actually a good replacement for most of the custom scripts you’ve seen me list here. When it comes to SNMP data, you can have Telegraf capture all of this and drop it into a InfluxDB. I haven’t spent a tremendous amount of time with it yet, and admittedly the last thing I setup was my Docker statistics. Knowing what I know now, I’ll likely be making more post consolidating the previously used scripts into the one Telegraf instances to capture all of the SNMP data I’m getting manually. For now, the scripts work very well (and are light on resources).
Install Telegraf with Docker
Create the docker volume storage location
# mkdir -p /monitoring/telegraf/
Generate the default configuration file
# docker run --rm telegraf -sample-config > /monitoring/telegraf/telegraf.conf
Modify this file to fit your needs, lots of modification at the top for your environment. In particular look for these two things.
Output Plugin
# Configuration for influxdb server to send metrics to [[outputs.influxdb]] urls = ["http://localhost:8086"] # required database = "telegraf" # required precision = "s" retention_policy = "default" write_consistency = "any"timeout = "5s"
Docker Configuration
[[inputs.docker]] ## Docker Endpoint ## To use TCP, set endpoint = "tcp://[ip]: [port]" ## To use environment variables (ie, docker-machine), set endpoint = "ENV" #endpoint = "unix:///var/run/docker.sock" #you may need to use the actual IP here endpoint = "tcp://localhost:2375" ## Only collect metrics for these containers, collect all if empty container_names = [] ## Timeout for docker list, info, and stats commands timeout = "30s"
Docker upgrade script
If you are looking for easy redeployment upgrade script for Telegraf container refer our repo, devops
Autostart the image – create a systemd startup script
# vi /lib/systemd/system/telegraf.service
[Unit] Description=telegraf container Requires=docker.service After=docker.service [Service] User=foxutech Restart=on-failure RestartSec=45Exec Start=/usr/bin/docker start -a telegraf ExecStop=/usr/bin/docker stop -t 2 telegraf [Install] WantedBy=multi-user.target
Enable on startup
# systemctl enable telegraf.service
If you want to start it up for testing
# systemctl start telegraf
Once telegraf starts, you should see data populating in your influxdb.
