Promtail¶

What Is It?¶

Promtail is an open-source log shipping agent developed by Grafana Labs, designed specifically to work with Loki. It runs alongside your services, tails log files (or the systemd journal), attaches metadata labels, and pushes log entries to a Loki instance.

Promtail is lightweight and easy to configure. In this course it runs as a Docker Compose service, mounting the host's /var/log directory and shipping system logs to Loki.

Installation¶

Promtail is deployed as a Docker Compose service. No dnf installation is needed.

Container image:

grafana/promtail:latest — log shipping agent

Key Files and Directories¶

promtail-config.yaml: Promtail configuration file. Defines where to read logs from, which labels to attach, and where to push them. Bind-mounted into the container at /etc/promtail/config.yaml.
/tmp/positions.yaml (inside container): Tracks the current read position in each tailed log file. Allows Promtail to resume after a restart without re-sending old entries.

Default Ports¶

Promtail does not expose any ports for external access. It only makes outbound connections to push logs to Loki.

Port	Protocol	Purpose
9080	TCP	Promtail HTTP status page (internal only)

Configuration¶

Minimal Working Configuration¶

promtail-config.yaml — place this file in your lab11 working directory:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/{messages,secure,cron,*.log}

The http://loki:3100 URL uses the Docker Compose service name for internal DNS resolution.

Docker Compose service definition (add to docker-compose.yml):

promtail:
  image: grafana/promtail:latest
  volumes:
    - ./promtail-config.yaml:/etc/promtail/config.yaml:ro
    - /var/log:/var/log:ro
  command: -config.file=/etc/promtail/config.yaml
  restart: unless-stopped

The host's /var/log is mounted read-only into the container so Promtail can access system log files.

Important Directives¶

clients.url: The Loki push endpoint. In Docker Compose, use the Loki service name: http://loki:3100/loki/api/v1/push.
scrape_configs: Defines which files to tail and what labels to assign. Each file set is a separate static_config entry.
labels.__path__: A glob pattern for the files to tail. Supports {a,b,c} brace expansion.
labels.job: A user-defined label value. The scoring check queries Loki for {job="varlogs"} — this label must match exactly.
positions.filename: Where Promtail saves its read position state. Use /tmp/positions.yaml so it's inside the container's writable tmpfs.

Common Commands¶

# Check Promtail status
sudo docker compose logs promtail

# Verify logs are flowing (query Loki API from the VM)
curl "http://localhost:3100/loki/api/v1/query?query={job=\"varlogs\"}&limit=5"

# Check what /var/log files are visible inside the container
sudo docker compose exec promtail ls /var/log/

# Tail Promtail logs for live debugging
sudo docker compose logs -f promtail

Logging and Debugging¶

Promtail starts but no logs in Loki: check that /var/log/messages exists on the host and is readable. On CentOS Stream, messages and secure are the main syslog files.
connection refused to Loki: Promtail starts before Loki is ready. Add depends_on: [loki] to the Promtail service in docker-compose.yml.
entry out of order errors: Loki received log entries with timestamps older than the reject_old_samples_max_age. Check the Loki config.
Positions file lost on restart: normal if using /tmp/positions.yaml — Promtail re-reads the tail of each file on startup.

Troubleshooting checklist:

sudo docker compose ps — is the promtail container running?
sudo docker compose logs promtail — any startup errors or connection failures?
curl "http://localhost:3100/loki/api/v1/query?query={job=\"varlogs\"}&limit=1" — are logs reaching Loki?
Does /var/log/messages exist on the host? (ls -la /var/log/messages)

Security Considerations¶

Read-only log access: Promtail mounts /var/log read-only — it cannot modify system logs.
No external port: Promtail does not expose any port externally, so no firewall changes are needed for it to function.
Log content: system logs may contain sensitive information. Ensure Loki's port 3100 is firewalled from untrusted networks.