Host & Container Monitoring with Prometheus

Monitoring your systems and containers is essential for maintaining a reliable homelab or home server. A popular setup involves Prometheus, Node Exporter, and cAdvisor for collecting metrics, combined with Grafana for creating insightful dashboards.

In this guide, we’ll set up a complete monitoring solution by:

1. Configuring Prometheus to scrape metrics from Node Exporter and cAdvisor.
2. Using Grafana to visualize the data with intuitive dashboards.

Let’s dive in and build a robust monitoring stack!

Setup Node Exporter & Cadvisor

To organize and store configuration files for monitoring, create dedicated folders for Node Exporter and cAdvisor.

Run the following commands to create the required directories:

1mkdir nodeexporter
2mkdir cadvisor

Next, we’ll set up a docker-compose.yml` file in each folder to configure the respective services.

Node Exporter

Open a new docker-compose.yml file for editing:

1nano nodeexporter/docker-compose.yml

Paste the following content into the file:

 1services:
 2  nodeexporter:
 3    image: prom/node-exporter
 4    container_name: nodeexporter
 5    volumes:
 6      - /proc:/host/proc:ro
 7      - /sys:/host/sys:ro
 8      - /:/rootfs:ro
 9    command:
10      - '--path.procfs=/host/proc'
11      - '--path.rootfs=/rootfs'
12      - '--path.sysfs=/host/sys'
13      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
14    restart: unless-stopped
15    network_mode: host

The network_mode: host setting allows Node Exporter to access the host network interfaces, enabling it to collect networking metrics.

cAdvisor

Open a new docker-compose.yml file for editing:

1nano cadvisor/docker-compose.yml

Paste the following content into the file:

 1services:
 2  cadvisor:
 3    image: gcr.io/cadvisor/cadvisor
 4    container_name: cadvisor
 5    privileged: true
 6    devices:
 7      - /dev/kmsg:/dev/kmsg
 8    environment:
 9      - TZ=Europe/Amsterdam
10    volumes:
11      - /:/rootfs:ro
12      - /var/run:/var/run:ro
13      - /sys:/sys:ro
14      - /var/lib/docker:/var/lib/docker:ro
15      - /cgroup:/cgroup:ro
16    command: 
17      - '--housekeeping_interval=15s'
18      - '--docker_only=true'
19    restart: unless-stopped
20    networks:
21      - backend
22networks:
23  backend:
24    name: backend

Now that we have the configurations in place, we can start Node Exporter and cAdvisor by running the following commands:

1docker compose -f nodeexporter/docker-compose.yml up -d
2docker compose -f cadvisor/docker-compose.yml up -d

Setup Prometheus

To collect the metrics from Node Exporter and cAdvisor, we’ll create a dedicated directory for Prometheus to store its configuration and Docker Compose files.

First, create the Prometheus folder:

1mkdir prometheus

Next, create a docker-compose.yml file for Prometheus:

1nano prometheus/docker-compose.yml

Add the following configuration to the file:

 1services:
 2  prometheus:
 3    image: prom/prometheus
 4    container_name: prometheus
 5    environment:
 6      - TZ=Europe/Amsterdam
 7    volumes:
 8      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
 9      - prometheus:/prometheus
10    command:
11      - '--config.file=/etc/prometheus/prometheus.yml'
12      - '--storage.tsdb.path=/prometheus'
13      - '--web.console.libraries=/etc/prometheus/console_libraries'
14      - '--web.console.templates=/etc/prometheus/consoles'
15      - '--storage.tsdb.retention.size=100GB'
16      - '--web.enable-lifecycle'
17    restart: unless-stopped
18    expose:
19      - 9090
20    networks:
21      - backend
22    extra_hosts:       
23      - "host.docker.internal:host-gateway"
24networks:
25  backend:
26    name: backend
27volumes:
28    prometheus:
29      name: prometheus

Prometheus requires a configuration file to define which services to scrape for metrics. Create the configuration file:

1nano prometheus/prometheus.yml

Add the following configuration to the file:

 1global:
 2  scrape_interval:     15s
 3  evaluation_interval: 15s
 4scrape_configs:
 5  - job_name: 'cadvisor'
 6    scrape_interval: 10s
 7    static_configs:
 8      - targets: ['cadvisor:8080']
 9  - job_name: 'prometheus'
10    scrape_interval: 10s
11    static_configs:
12      - targets: ['localhost:9090']
13  - job_name: 'nodeexporter'
14    scrape_interval: 10s
15    static_configs:
16      - targets: ['hostip:9100']
17    metric_relabel_configs:
18      - source_labels: [nodename]
19        target_label: "instance"
20        action: "replace"

Since Node Exporter is using the host network, you need to replace hostip with your actual host IP address. The metric_relabel_configs will help in relabeling the hostip:9100 to the actual hostname, making it easier to identify.

Now that you have configured Prometheus, you can start it with the following command:

1docker compose -f prometheus/docker-compose.yml up -d

This command starts the Prometheus container, which will begin collecting metrics from both Node Exporter and cAdvisor, providing comprehensive monitoring for your systems and containers.

Setup Grafana

To finalize your monitoring setup, we’ll create a directory for Grafana to store its Docker Compose and configuration files.

First, create the Grafana folder:

1mkdir grafana

Next, create a docker-compose.yml file for Grafana:

1nano grafana/docker-compose.yml

Add the following configuration to the file:

 1services:
 2  grafana:
 3    image: grafana/grafana
 4    container_name: grafana
 5    environment:
 6      - TZ=Europe/Amsterdam
 7    volumes:
 8      - grafana_data:/var/lib/grafana
 9    restart: unless-stopped
10    ports:
11      - 3000:3000
12    networks:
13      - backend
14networks:
15  backend:
16    name: backend
17volumes:
18    grafana_data:
19      name: grafana_data

Now you can start Grafana by running:

1docker compose -f grafana/docker-compose.yml up -d

Once Grafana is running, open your browser and navigate to: http://<HOST_IP>:3000

• Default login credentials:
  • Username: admin
  • Password: admin

Datasource

To visualize the data collected by Prometheus, you need to add it as a data source in Grafana:

1. Click Connections in the left-side menu.
2. Search for Prometheus
3. Click Add new Datasource
4. Enter the name prometheus
5. Fill in the Prometheus server URL http://prometheus:9090

Dashboards

To see all the metrics we need to dashboards. You can make your own dashboards or use mine as a starter:

• NodeExporter
• cAdvisor

Conclusion

Congratulations! You have successfully set up host and container monitoring with Prometheus and Grafana. Your monitoring system is now capable of visualizing the metrics from your applications and self-hosting.