Grafana Observability: Building the Stack

14 min
2960 words
Categories: Monitoring
Tags: Docker Grafana Prometheus Loki Alloy Systemd

Before you can collect metrics or centralize logs, you need somewhere to store and visualize them — and an agent to collect the data. This post sets up all four components of a Grafana observability stack:

  • Prometheus — metrics storage
  • Loki — log storage
  • Grafana — visualization
  • Grafana Alloy — telemetry collection agent

Prometheus

Prometheus is a time-series database that stores metrics. We’ll enable the remote write receiver so Alloy can push metrics directly to it.

Create a directory and Docker Compose file:

BASH
mkdir prometheus
nano prometheus/docker-compose.yml
docker-compose.yml
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    environment:
      - TZ=Europe/Amsterdam
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=90d'
      - '--storage.tsdb.retention.size=100GB'
      - '--web.enable-lifecycle'
      - '--web.enable-remote-write-receiver'
    restart: unless-stopped
    networks:
      - backend

networks:
  backend:
    name: backend

volumes:
  prometheus:
    name: prometheus

Create the Prometheus config file:

BASH
nano prometheus/prometheus.yml
prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs: []

scrape_configs is empty because Alloy pushes metrics via remote write. Adjust retention.time and retention.size to match your available disk space — whichever limit is hit first triggers cleanup.

Start Prometheus:

BASH
docker compose -f prometheus/docker-compose.yml up -d

Verify it’s ready:

BASH
curl -s http://localhost:9090/-/ready

Loki

Loki stores logs indexed by labels rather than full-text, making it storage-efficient and fast for label-based queries.

Create a directory and Docker Compose file:

BASH
mkdir loki
nano loki/docker-compose.yml
docker-compose.yml
services:
  loki:
    image: grafana/loki
    container_name: loki
    restart: unless-stopped
    environment:
      - TZ=Europe/Amsterdam
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
      - loki-data:/loki
    command: -config.file=/etc/loki/loki-config.yaml
    networks:
      - backend

networks:
  backend:
    name: backend

volumes:
  loki-data:
    name: loki-data

Create the Loki config file:

BASH
nano loki/loki-config.yaml
loki-config.yaml
auth_enabled: false

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: info

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: memberlist

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  query_timeout: 600s
  retention_period: "365d"
  ingestion_rate_mb: 4
  ingestion_burst_size_mb: 6
  max_streams_per_user: 10000
  max_line_size: 256000
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  creation_grace_period: 15m
  discover_log_levels: false

Adjust retention_period based on your available disk space.

Start Loki:

BASH
docker compose -f loki/docker-compose.yml up -d

Verify it’s ready:

BASH
curl -s http://localhost:3100/ready

Grafana

Grafana is the visualization layer — it connects to Prometheus and Loki and lets you build dashboards and explore data.

Create a directory and Docker Compose file:

BASH
mkdir grafana
nano grafana/docker-compose.yml
docker-compose.yml
services:
  grafana:
    image: grafana/grafana
    container_name: grafana
    hostname: ${HOSTNAME}
    environment:
      - TZ=Europe/Amsterdam
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
    restart: unless-stopped
    ports:
      - 3000:3000
    networks:
      - backend

networks:
  backend:
    name: backend

volumes:
  grafana_data:
    name: grafana_data

Change GF_SECURITY_ADMIN_PASSWORD to a secure password before starting.

Start Grafana:

BASH
docker compose -f grafana/docker-compose.yml up -d

Open http://<HOST_IP>:3000 and log in with username admin and your password.

Add Datasources

Connect Grafana to Prometheus and Loki:

Prometheus:

  1. Click Connections → search for Prometheus → Add new Datasource
  2. Set name prometheus, URL http://prometheus:9090
  3. Click Save & Test

Loki:

  1. Click Connections → search for Loki → Add new Datasource
  2. Set name loki, URL http://loki:3100
  3. Click Save & Test

Grafana Alloy

Alloy is the telemetry collection agent that replaces Promtail and Grafana Agent. It collects metrics, logs, and traces and forwards them to Prometheus and Loki using a modular config — each collector lives in its own .alloy file, and every file in the config directory is loaded automatically.

You can run Alloy two ways, and both are covered below:

  • Docker — simplest if the rest of your stack is already containerized on the same host
  • systemd — installs Alloy directly on the host, giving it native access to resources like /var/log or the Docker socket without volume mounts, and lets it start earlier in the boot sequence than Docker itself

Alloy (Docker)

Create a directory for Alloy and its config files:

BASH
mkdir -p alloy/config
nano alloy/docker-compose.yml
docker-compose.yml
services:
  alloy:
    image: grafana/alloy:latest
    container_name: alloy
    hostname: ${HOSTNAME}
    restart: unless-stopped
    environment:
      - TZ=Europe/Amsterdam
    ports:
      - "12345:12345"
    volumes:
      - ./config/:/etc/alloy/config/:ro
      - alloy-data:/var/lib/alloy/data
    command:
      - run
      - --server.http.listen-addr=0.0.0.0:12345
      - --storage.path=/var/lib/alloy/data
      - /etc/alloy/config/
    networks:
      - backend

networks:
  backend:
    name: backend

volumes:
  alloy-data:
    name: alloy-data

Alloy loads every .alloy file in the config/ directory automatically — adding a new collector is as simple as dropping in a new file. Port 12345 is the Alloy web UI for debugging component status.

Endpoints

Create endpoint.alloy to centralize write destinations. All collector configs reference these by name:

BASH
nano alloy/config/endpoint.alloy
endpoint.alloy
loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://prometheus:9090/api/v1/write"
  }
}

Self-Monitoring

Create self.alloy so Alloy reports its own health metrics to Prometheus:

BASH
nano alloy/config/self.alloy
self.alloy
prometheus.exporter.self "alloy_metrics" {}

prometheus.scrape "alloy_metrics" {
  targets         = prometheus.exporter.self.alloy_metrics.targets
  scrape_interval = "60s"
  forward_to      = [prometheus.remote_write.default.receiver]
}

Start Alloy:

BASH
docker compose -f alloy/docker-compose.yml up -d

Open the Alloy web UI at http://<HOST_IP>:12345 to verify all components are green.

Alloy (systemd)

Running Alloy as a systemd service is an alternative to the Docker container above — useful when you want the agent to have direct access to the host filesystem without volume mounts, or to have it start earlier in the boot process than Docker.

Install

Add the Grafana apt repository and install the alloy package:

BASH
sudo apt-get install -y apt-transport-https software-properties-common wget

sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | \
  sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install -y alloy

The package creates an alloy system user, installs the binary at /usr/bin/alloy, and registers a systemd unit. It does not start automatically after install.

Configure

The systemd unit reads startup options from /etc/default/alloy. Edit it to load a config directory instead of a single file — the same pattern used in the Docker setup:

BASH
sudo nano /etc/default/alloy
/etc/default/alloy
CONFIG_FILE="/etc/alloy/config/"
CUSTOM_ARGS="--server.http.listen-addr=0.0.0.0:12345"
STATE_DIRECTORY="/var/lib/alloy"

By default Alloy only listens on localhost:12345, so CUSTOM_ARGS is used here to expose the web UI on all interfaces. This is optional.

Create the config directory:

BASH
sudo mkdir -p /etc/alloy/config

The endpoint.alloy and self.alloy files are the same config as the Docker path above — copy them into /etc/alloy/config/. The one difference: since the systemd service isn’t on the Docker backend network, replace the loki and prometheus hostnames in endpoint.alloy with the actual IP or hostname of your Prometheus and Loki instances, e.g. http://<HOST>:3100/loki/api/v1/push and http://<HOST>:9090/api/v1/write.

Fix permissions so the config files are readable by the alloy user:

BASH
sudo chown -R alloy:alloy /etc/alloy/config
sudo chmod -R 750 /etc/alloy/config

Permissions

The alloy user only has access to its own files by default. If you plan to collect system logs or Docker container stats and logs, you need to grant it read access to those resources.

System logs (/var/log):

BASH
sudo usermod -aG adm alloy

Docker container logs and stats:

Adding the alloy user to the docker group is not enough — Docker’s cgroup files and log directories are owned by root and require CAP_DAC_READ_SEARCH to traverse. Create a systemd drop-in to grant that capability:

BASH
sudo mkdir -p /etc/systemd/system/alloy.service.d
sudo tee /etc/systemd/system/alloy.service.d/capabilities.conf <<'EOF'
[Service]
AmbientCapabilities=CAP_DAC_READ_SEARCH
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
EOF
sudo systemctl daemon-reload
sudo usermod -aG docker alloy

A service restart is required after any permission change.

Start

Enable and start the service:

BASH
sudo systemctl enable alloy
sudo systemctl start alloy

Check that it came up cleanly:

BASH
sudo systemctl status alloy

You should see active (running). If it failed, check the journal:

BASH
sudo journalctl -u alloy -n 50

Verify

Confirm Alloy is healthy and listening:

BASH
curl -s http://localhost:12345/-/healthy

Open the Alloy web UI at http://<HOST_IP>:12345 and verify all components show green. The prometheus.remote_write.default and loki.write.default components should both be running.

With the full stack running — whether Alloy is deployed as a Docker container or a systemd service — it’s time to start adding collectors. Alloy’s built-in exporters and log collectors gather host metrics, Docker container metrics, system logs, and Docker container logs, all through the single agent instead of separate containers per exporter. Metrics land in Prometheus, logs land in Loki, and both are visualized in Grafana.

Host Metrics

Create an Alloy config file for host system metrics:

BASH
nano alloy/config/unix.alloy
unix.alloy
prometheus.exporter.unix "unix" {
  rootfs_path = "/rootfs"
  procfs_path = "/rootfs/proc"
  sysfs_path  = "/rootfs/sys"
  disable_collectors = ["ipvs", "btrfs", "infiniband", "xfs", "zfs"]
  enable_collectors  = ["meminfo", "processes"]

  filesystem {
    fs_types_exclude     = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
    mount_points_exclude = "^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)"
    mount_timeout        = "5s"
  }

  netclass {
    ignored_devices = "^(veth.*|cali.*|[a-f0-9]{15})$"
  }

  netdev {
    device_exclude = "^(veth.*|cali.*|[a-f0-9]{15})$"
  }
}

prometheus.scrape "unix" {
  targets    = prometheus.exporter.unix.unix.targets
  forward_to = [prometheus.remote_write.default.receiver]
}

Because Alloy runs inside Docker with the host filesystem bind-mounted at /rootfs, the rootfs_path, procfs_path, and sysfs_path fields tell the exporter where to find real system data rather than the container’s own filesystem.

disable_collectors turns off collectors you are unlikely to need on a typical Linux homelab (IPVS load balancer, Btrfs, InfiniBand). enable_collectors adds meminfo and processes, which are not enabled by default.

The filesystem block excludes virtual filesystem types (tmpfs, cgroup, overlay, etc.) and Docker-internal mount points so only real disks appear in Grafana. mount_timeout prevents the scrape from hanging if a network mount is unresponsive.

Both netclass and netdev use the same pattern to ignore virtual Ethernet interfaces created by Docker and Calico, as well as the 15-character hex interface names Kubernetes generates — keeping the network panels focused on real physical or VLAN interfaces.

This exporter needs the host filesystem mounted read-only into the Alloy container — see Apply & Verify below for the required volume mounts.

Container Metrics

Create an Alloy config file for Docker container metrics:

BASH
nano alloy/config/docker-metrics.alloy
docker-metrics.alloy
// Docker container metrics (CPU, memory, network per container)
prometheus.exporter.cadvisor "dockermetrics" {
  docker_host      = "unix:///var/run/docker.sock"
  storage_duration = "5m"
}

prometheus.relabel "docker_filter" {
  forward_to = [prometheus.remote_write.default.receiver]

  rule {
    target_label = "job"
    replacement  = "docker"
  }
  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }
  // Drop container_spec metrics that frequently contain NaN values
  rule {
    source_labels = ["__name__"]
    regex         = "container_spec_(cpu_period|cpu_quota|cpu_shares|memory_limit_bytes|memory_swap_limit_bytes|memory_reservation_limit_bytes)"
    action        = "drop"
  }
}

prometheus.scrape "dockermetrics" {
  targets         = prometheus.exporter.cadvisor.dockermetrics.targets
  forward_to      = [prometheus.relabel.docker_filter.receiver]
  scrape_interval = "10s"
}

This exporter needs the Docker socket mounted into the Alloy container — see Apply & Verify below.

System Logs

Centralized logging complements metrics by letting you search and correlate events across your infrastructure. Create an Alloy config file to collect auth.log and syslog from the host:

BASH
nano alloy/config/syslog.alloy
syslog.alloy
// System auth.log collection
local.file_match "authlog" {
  path_targets = [{
    __path__ = "/var/log/auth.log",
  }]
}

loki.source.file "authlog" {
  targets    = local.file_match.authlog.targets
  forward_to = [loki.process.authlog.receiver]
}

loki.process "authlog" {
  stage.static_labels {
    values = {
      job = "authlog",
    }
  }
  forward_to = [loki.write.default.receiver]
}

// System syslog collection
local.file_match "syslog" {
  path_targets = [{
    __path__ = "/var/log/syslog",
  }]
}

loki.source.file "syslog" {
  targets    = local.file_match.syslog.targets
  forward_to = [loki.process.syslog.receiver]
}

loki.process "syslog" {
  stage.static_labels {
    values = {
      job = "syslog",
    }
  }
  forward_to = [loki.write.default.receiver]
}

The path_targets field supports wildcards — you can use /var/log/*.log to collect all log files in a directory at once.

This collector needs the host’s /var/log directory mounted into the Alloy container — see Apply & Verify below.

Docker Container Logs

Alloy can automatically discover and collect logs from all running Docker containers — no manual configuration needed per container.

Create an Alloy config file for Docker log collection:

BASH
nano alloy/config/docker-logs.alloy
docker-logs.alloy
// Discover all running Docker containers
discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

// Extract useful labels from container metadata
discovery.relabel "containers" {
  targets = discovery.docker.containers.targets

  rule {
    source_labels = ["__meta_docker_container_name"]
    target_label  = "container"
    regex         = "/(.*)"
    replacement   = "$1"
  }

  rule {
    source_labels = ["__meta_docker_container_log_stream"]
    target_label  = "stream"
  }
}

// Collect logs from discovered containers
loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  forward_to = [loki.process.containers.receiver]
}

// Add a static job label and forward to Loki
loki.process "containers" {
  stage.static_labels {
    values = {
      job = "docker",
    }
  }
  forward_to = [loki.write.default.receiver]
}

This automatically picks up containers as they start and stop — no Alloy restart required once it’s running. It reuses the same Docker socket mount as the container metrics collector above, so no additional volume mount is needed for this one.

Apply & Verify

All four collectors above need volume mounts added to your Alloy docker-compose.yml:

docker-compose.yml
volumes:
  - /:/rootfs:ro
  - /sys:/sys:ro
  - /run/udev/data:/run/udev/data:ro
  - /var/run/docker.sock:/var/run/docker.sock:ro
  - /var/log:/var/log:ro

Recreate the Alloy container to pick up both the new mounts and the new config files in one step:

BASH
docker compose up -d alloy

Open the Alloy web UI and confirm all four components are healthy:

  • prometheus.exporter.unix.unix
  • prometheus.exporter.cadvisor.dockermetrics
  • loki.source.file.authlog / loki.source.file.syslog
  • loki.source.docker.containers

Then verify data is flowing in Grafana’s Explore view — metrics first:

PROMQL
node_cpu_seconds_total{job="unix"}
PROMQL
container_cpu_usage_seconds_total{job="docker"}

Then logs:

LOGQL
{job="syslog"}
LOGQL
{job="docker"}

Filter by container name for more targeted queries:

LOGQL
{job="docker", container="traefik"}

Grafana Dashboards

You can create your own dashboards or use these as a starting point:

License

Author: Sven van Ginkel

Link: https://svenvg.com/posts/grafana-observability-building-the-stack/

License: CC BY-NC-SA 4.0

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Please attribute the source, use non-commercially, and maintain the same license.

Related content