DIYLABHub.com

Monitoring Docker containers is essential for maintaining healthy, high-performing containerized applications. This guide will walk you through setting up a robust monitoring system using Prometheus and Grafana, two powerful open-source tools that work seamlessly together to collect, store, and visualize metrics from your Docker environment.

🔹Understanding Container Monitoring Architecture

Before diving into the setup, it’s important to understand how the monitoring stack works. In a typical Docker monitoring setup with Prometheus and Grafana, the architecture consists of several components working together:

Prometheus serves as the central metrics collection and storage system. It scrapes metrics from various exporters at regular intervals (typically every 15 seconds) and stores them in its time-series database. Grafana connects to Prometheus as a data source and creates visualizations through dashboards. These components work with specialized exporters that collect specific metrics.

🔹 Prerequisites

Before starting the setup process, ensure you have the following installed on your system:

• Docker Engine (latest stable version)
• Docker Compose
• Basic understanding of YAML configuration files
• Access to terminal/command line interface

🔹Setting Up Your Docker Monitoring Stack

1️⃣ Create Directory Structure

First, establish a proper directory structure to organize your configuration files:

mkdir -p docker-monitoring/prometheus
mkdir -p docker-monitoring/grafana/provisioning/datasources
mkdir -p docker-monitoring/grafana/provisioning/dashboards
cd docker-monitoring

2️⃣ Create Prometheus Configuration

Create a file named prometheus.yml in the prometheus directory:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

This configuration tells Prometheus to scrape metrics from itself, cAdvisor (for container metrics), and Node Exporter (for host metrics) every 15 seconds.

3️⃣ Create Docker Compose File

Create a docker-compose.yml file in the docker-monitoring directory:

version: '3.8'

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data: {}
  grafana_data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    ports:
      - "9090:9090"
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($|/)'
    ports:
      - "9100:9100"
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "8080:8080"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    ports:
      - "3000:3000"
    networks:
      - monitoring
    depends_on:
      - prometheus

This Docker Compose file defines four services: Prometheus, Node Exporter, cAdvisor, and Grafana. Each service is configured with appropriate volumes, ports, and network settings.

4️⃣ Launch the Monitoring Stack

Start all the services using Docker Compose:

docker-compose up -d

This command launches all four containers in detached mode. You can verify that they’re running properly with:

docker-compose ps

🔹 Configuring Grafana

1️⃣ Access Grafana Dashboard

Open your web browser and navigate to http://localhost:3000. You’ll be greeted with the Grafana login page. Use the following credentials:

• Username: admin
• Password: admin

You’ll be prompted to change the password on your first login for security purposes.

2️⃣ Add Prometheus as a Data Source

1. Click on the gear icon (⚙️) in the left sidebar to open the Configuration menu
2. Select “Data sources”
3. Click “Add data source”
4. Select “Prometheus” from the list
5. Set the URL field to http://prometheus:9090
6. Click “Save & Test” to verify the connection

If everything is set up correctly, you should see a green “Data source is working” message.

3️⃣ Import Dashboards

1. Click the “+” icon in the left sidebar
2. Select “Import”
3. Enter one of these dashboard IDs:
   • 893 (Docker and system monitoring)
   • 10619 (Docker containers)
   • 1860 (Node Exporter Full)
4. Click “Load”
5. Select your Prometheus data source from the dropdown
6. Click “Import”

These dashboards provide comprehensive monitoring for your Docker containers and host system metrics.

🔹Monitoring Multiple Hosts

If you need to monitor Docker containers across multiple hosts, you’ll need to install Node Exporter and cAdvisor on each host machine, then update your Prometheus configuration to scrape metrics from these additional targets.

Update Prometheus Configuration

Modify your prometheus.yml file to include the additional hosts:

scrape_configs:
  # Existing jobs...

  - job_name: 'node-exporter-host2'
    static_configs:
      - targets: ['host2-ip:9100']

  - job_name: 'cadvisor-host2'
    static_configs:
      - targets: ['host2-ip:8080']

After updating the configuration, reload Prometheus:

curl -X POST http://localhost:9090/-/reload

🔹Common Issues and Troubleshooting

🔹 Advanced Tips and Best Practices

💾 Persistence and Data Retention

To ensure your metrics data persists across container restarts, use named volumes for Prometheus and Grafana as shown in the Docker Compose file. You can adjust Prometheus’s data retention period by adding this to the command section:

command:
  # Existing commands...
  - '--storage.tsdb.retention.time=15d'

This sets data retention to 15 days, but you can adjust based on your storage capacity and requirements.

🔒 Security Considerations

1. Change default credentials immediately after setup
2. Consider using environment variables for sensitive information rather than hardcoding in files
3. Restrict access to the Prometheus and Grafana interfaces using a reverse proxy with authentication
4. Use network segmentation to isolate your monitoring stack from production services

⚡ Performance Optimization

1. Adjust scrape intervals based on your needs – more frequent scraping provides higher resolution but increases resource usage
2. Use recording rules in Prometheus for frequently used or complex queries
3. Consider federation for large-scale deployments
4. Monitor the resource usage of your monitoring stack itself

🔔 Adding Alerts

While this guide focuses on monitoring, you can extend your setup to include alerting:

1. Define alerting rules in Prometheus
2. Set up Alertmanager alongside Prometheus
3. Configure notification channels in Grafana
4. Create alert thresholds for critical metrics

🔹 Conclusion

You now have a fully functional Docker monitoring stack with Prometheus and Grafana. This setup provides comprehensive visibility into the performance and health of your Docker containers and host system. By following the tips and best practices outlined in this guide, you can ensure your monitoring system remains reliable, secure, and efficient.

Remember that monitoring is an ongoing process. As your containerized applications evolve, you may need to adjust your monitoring strategy by adding more exporters, creating custom dashboards, or implementing automated alerting based on your specific requirements.

🔹 References

1. Reddit: Getting started Grafana Prometheus monitoring
2. Reddit: What grafana dashboard you use to monitor docker
3. Reddit: How do I add Prometheus as a datasource to Grafana
4. Reddit: Docker monitoring with grafana and adding second host
5. Building a Docker-based Monitoring System with Prometheus and Grafana
6. Reddit: Prometheus not working after configuring docker named volumes
7. Reddit: Monitoring docker-containers with grafana, prometheus & fluent-bit
8. Reddit: Problems with Grafana and Prometheus Dashboard
9. Reddit: The Essential Guide to Grafana Docker Monitoring
10. Server Fault: Still can’t connect from my grafana docker container to prometheus
11. Kubernetes Training: Prometheus & Grafana: Docker Compose Monitoring Tutorial
12. Grafana Community: Connection error for data source Prometheus with grafana
13. Dev.to: How to Set Up Grafana and Prometheus Using Docker
14. Grafana Community: Tutorial Grafana Fundamentals Docker Prometheus
15. GitHub: Docker Monitoring Workshop with Grafana Alloy, Prometheus and Loki
16. YouTube: Install Grafana, Prometheus & Node Exporter with Docker