Monitor Docker Host Environment using Prometheus

Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

Prometheus is a time-series database with a UI and sophisticated querying language (PromQL). Prometheus can scrape metrics, counters, gauges and histograms over HTTP using plaintext or a more efficient protocol.

Prometheus’ main distinguishing features as compared to other monitoring systems are:

• a multi-dimensional data model (time series defined by metric name and set of key/value dimensions)
• a flexible query language to leverage this dimensional
• no dependency on distributed storage; single server nodes are autonomous
• timeseries collection happens via a pull model over HTTP
• pushing timeseries is supported via an intermediary gateway
• targets are discovered via service discovery or static configuration
• multiple modes of graphing and dash boarding support
• support for hierarchical and horizontal federation

When the /metrics endpoint is embedded within an existing application it’s referred to as instrumentation and when the /metrics endpoint is part of a stand-alone process the project call that an Exporter.

Node exporter

One of the most widely used exporters is the NodeExporter. When NodeExporter is run on a host it will provide details on I/O, memory, disk and CPU pressure. You can run the NodeExporter as a Docker container, but it needs so many additional flags that the project recommends you run it directly on a host being monitored.

Run Node Exporter in Docker

Let’s define a Docker Compose which will let us keep our command-lines simple and repeatable:

node-exporter:
   image: prom/node-exporter
   expose:
      - 9100

The built-in exporter

Prometheus provides its own set of metrics – in effect dog-fooding. This is built-in and is usually configured to be scraped (or collected) by default.

cAdvisor

cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running containers. It is a running daemon that collects, aggregates, processes, and exports information about running containers. Specifically, for each container it keeps resource isolation parameters, historical resource usage, histograms of complete historical resource usage and network statistics. This data is exported by container and machine-wide.

cAdvisor has native support for Docker containers and should support just about any other container type out of the box. We strive for support across the board so feel free to open an issue if that is not the case. cAdvisor’s container abstraction is based on lmctfy‘s so containers are inherently nested hierarchically.

Run cAdvisor in Docker

Let’s define a Docker Compose which will let us keep our command-lines simple and repeatable:

cadvisor:
   image: google/cadvisor
   volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
   expose:
      - 8080

Now we can just browse the monitoring graphs at http://localhost:8080.

Getting Prometheus with Docker

Prometheus is written in Golang and can be consumed as single statically-compiled binary with no other dependencies. The project also packages up the binary with a sensible configuration in a Docker container.

Run Prometheus in Docker

Let’s define a Docker Compose which will let us keep our command-lines simple and repeatable:

prometheus:
   image: prom/prometheus
   container_name: prometheus
   volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
   command:
      - '-config.file=/etc/prometheus/prometheus.yml'
      - '-storage.local.path=/prometheus'
      - '-alertmanager.url=http://alertmanager:9093'
   expose:
      - 9090
   ports:
      - 9090:9090

Navigate to: http://localhost:9090/ to view the UI.

In the screenshot above you can see the amount of go_routines being used as recorded by Prometheus itself. To see the raw metrics Prometheus produces about itself open a browser and head over to http://localhost:9090/metrics

A Go routine is a light-weight version of a thread; they’re used in Golang to enable concurrency.

Read More: Monitor Docker Environment using InfluxDB, Grafana & Telegraf

You may not want to monitor Prometheus, but the fact that batteries are included means you can get a feel for metrics from the get go.

How to setup this Environment:

Make sure you installed Docker and docker-compose in your server. Then clone this repository and run docker-compose as mentioned below

# git clone https://github.com/foxutech/Prometheus.git
# cd Prometheus/Prometheus-Monitoring
# docker-compose up –d

To make sure all the containers are up and running

# docker ps or docker-compose ps