This document outlines the monitoring stack used to observe the Docker server's performance, logs, and health.
The monitoring stack uses the following components for metrics collection:
Log collection is handled by:
Prometheus scrapes metrics from various exporters and stores them in a time series database.
Configuration:
/home/ali3nz/docker/monitoring/prometheus/prometheus.ymlglobal:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node'
static_configs:
- targets: ['node-exporter:9100']
- job_name: 'fortigate'
static_configs:
- targets: ['fortigate-exporter:9116']
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
Node Exporter collects system-level metrics such as:
Loki stores and indexes logs, while Promtail collects and forwards them to Loki.
Promtail Configuration:
/home/ali3nz/docker/monitoring/promtail/config.ymlserver:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'container'
Grafana provides dashboards and visualizations for all collected metrics and logs.
Alert Manager handles alert routing, grouping, and notifications.
Configuration:
/home/ali3nz/docker/monitoring/alertmanager/alertmanager.ymlglobal:
resolve_timeout: 5m
route:
group_by: ['alertname', 'job']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'email'
receivers:
- name: 'email'
email_configs:
- to: '[email protected]'
from: '[email protected]'
smarthost: 'smtp.example.com:587'
auth_username: '[email protected]'
auth_password: 'password'
The monitoring stack includes several predefined dashboards:
Alerts are configured to notify for the following conditions:
Alert notifications are sent via:
To access the monitoring stack:
Regular maintenance tasks for the monitoring stack:
Common issues and resolutions:
Prometheus Not Collecting Data:
docker logs prometheus
# Check target status
curl -s http://localhost:9090/api/v1/targets | jq .
Loki Not Receiving Logs:
docker logs promtail
docker logs loki
Grafana Dashboard Issues:
docker logs grafana
# Restart if needed
docker restart grafana