Monitoring isn’t just about metrics—it’s about ensuring application health. Centralized logging with Loki and Grafana provides deeper insights by visualizing and searching logs, helping you quickly identify and resolve issues.
Setup Loki
To set up Loki, we need to create a folder to hold both the docker-compose.yml and the configuration file.
First, create the folder for Loki:
1mkdir lokiOpen a new docker-compose.yml file for editing:
1nano loki/docker-compose.ymlPaste the following content into the file:
1services:
2 loki:
3 image: grafana/loki
4 container_name: loki
5 restart: unless-stopped
6 environment:
7 - TZ=Europe/Amsterdam
8 expose:
9 - 3100
10 volumes:
11 - ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
12 - loki:/tmp
13 command: -config.file=/etc/loki/loki-config.yaml
14 networks:
15 - backend
16networks:
17 backend:
18 name: backend
19volumes:
20 loki:
21 name: lokiLoki requires a configuration file to define which services to scrape for metrics. Create the configuration file:
1nano loki/loki-config.yamlPaste the following content into the file:
1auth_enabled: false
2server:
3 http_listen_port: 3100
4 grpc_listen_port: 9096
5common:
6 instance_addr: 127.0.0.1
7 path_prefix: /tmp/loki
8 storage:
9 filesystem:
10 chunks_directory: /tmp/loki/chunks
11 rules_directory: /tmp/loki/rules
12 replication_factor: 1
13 ring:
14 kvstore:
15 store: inmemory
16schema_config:
17 configs:
18 - from: 2020-10-24
19 store: tsdb
20 object_store: filesystem
21 schema: v13
22 index:
23 prefix: index_
24 period: 24h
25query_range:
26 results_cache:
27 cache:
28 embedded_cache:
29 enabled: true
30 max_size_mb: 100
31querier:
32 max_concurrent: 500
33query_scheduler:
34 max_outstanding_requests_per_tenant: 1000
35frontend:
36 max_outstanding_per_tenant: 2000
37limits_config:
38 max_global_streams_per_user: 5000
39 ingestion_rate_mb: 50
40 per_stream_rate_limit: 50MBSetup Promtail
To finalize your logging setup with Loki, you’ll need to configure Promtail to send logs to Loki.
Start by creating a folder to store the docker-compose.yml and promtail-config.yaml files.
1mkdir promtailOpen a new docker-compose.yml file for editing:
1nano promtail/docker-compose.ymlPaste the following content into the file:
1services:
2 promtail:
3 image: grafana/promtail
4 container_name: promtail
5 restart: unless-stopped
6 environment:
7 - TZ=Europe/Amsterdam
8 volumes:
9 - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml:ro
10 - /var/log/:/logs
11 command: -config.file=/etc/promtail/promtail-config.yaml
12 networks:
13 - backend
14networks:
15 backend:
16 name: backendNow, create a configuration file named promtail-config.yaml:
1server:
2 http_listen_port: 9080
3 grpc_listen_port: 0
4positions:
5 filename: /tmp/positions.yaml
6clients:
7 - url: http://loki:3100/loki/api/v1/push
8scrape_configs:
9- job_name: authlog
10 static_configs:
11 - targets:
12 - authlog
13 labels:
14 job: authlog
15 __path__: /logs/auth.log
16- job_name: syslog
17 static_configs:
18 - targets:
19 - syslog
20 labels:
21 job: syslog
22 __path__: /logs/syslogThis configuration will scrape the system’s auth and syslog logs.
Note: You can customize the job_name, targets, job, and __path__ under scrape_configs according to your logging requirements.
Finally, start the Loki and Promtail services by running the following commands:
1docker compose -f loki/docker-compose.yml up -d
2docker compose -f promtail/docker-compose.yml up -dGrafana
To visualize logs from Loki in Grafana, you need to configure Loki as a datasource. Here’s how to do it:
- Open Grafana:
- Click Connections in the left-side menu.
- Search for Loki
- Click Add new Datasource
- Enter the name loki
- Fill in the Prometheus server URL
http://loki:3100
Exploring Logs in Grafana
Now that you have added Loki as a datasource, you can explore your logs:
- In the left sidebar, click on Explore.
- In the top-left dropdown menu, choose Loki as your datasource.
- In the query section, select the label filename and set the value to /logs/syslog
Summary
With Loki configured as a datasource in Grafana, Promtail will continuously send log files to Loki, allowing you to visualize and analyze logs easily. This setup provides a comprehensive monitoring solution, enabling you to monitor both metrics and logs from your applications.