Prometheus¶

What Is It?¶

Prometheus is an open-source monitoring and alerting toolkit that collects metrics from configured targets at regular intervals. It stores time-series data in its own database and provides a powerful query language called PromQL. It operates on a pull model — Prometheus periodically scrapes HTTP endpoints that expose metrics in the OpenMetrics text format.

node_exporter is a companion tool that exposes Linux host-level metrics (CPU, memory, disk, network) from the /proc and /sys filesystems in a format Prometheus can scrape.

Installation¶

Both Prometheus and node_exporter are deployed as Docker Compose services. No dnf installation is needed.

Container images used:

prom/node-exporter:latest — exposes host-level metrics on port 9100
prom/prometheus:latest — scrapes targets and stores metrics on port 9090

Key Files and Directories¶

prometheus.yml: The Prometheus configuration file. Defines scrape intervals and the list of targets to scrape. Bind-mounted into the container at /etc/prometheus/prometheus.yml.
/prometheus (inside container): Prometheus time-series database storage path. Backed by a named Docker volume for persistence.

Default Ports¶

Port	Protocol	Purpose
9090	TCP	Prometheus — web UI and API
9100	TCP	Node Exporter — host metrics

Configuration¶

Minimal Working Configuration¶

prometheus.yml — place this file in your lab11 working directory:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

The target node-exporter:9100 uses the Docker Compose service name for internal DNS resolution.

Docker Compose service definitions (add to docker-compose.yml):

node-exporter:
  image: prom/node-exporter:latest
  volumes:
    - /proc:/host/proc:ro
    - /sys:/host/sys:ro
    - /:/rootfs:ro
  command:
    - '--path.procfs=/host/proc'
    - '--path.sysfs=/host/sys'
    - '--path.rootfs=/rootfs'
    - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
  ports:
    - "9100:9100"
  restart: unless-stopped

prometheus:
  image: prom/prometheus:latest
  volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - prometheus-data:/prometheus
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
  ports:
    - "9090:9090"
  restart: unless-stopped

node_exporter needs read-only access to /proc, /sys, and / to collect host metrics.

Open firewall ports so the scoring server can reach both services:

sudo firewall-cmd --permanent --add-port=9100/tcp
sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --reload

Important Directives¶

scrape_interval: How often Prometheus fetches metrics from targets. 15s is a good default.
scrape_configs: List of scrape jobs. Each job has a name and a list of targets in host:port format.
static_configs.targets: Explicit list of endpoints to scrape. Within Docker Compose, use the service name as the hostname.
OpenMetrics format: Metrics are plain text: <metric_name>{<labels>} <value>. For example: node_cpu_seconds_total{cpu="0",mode="idle"} 12345.6.

Common Commands¶

# Verify node_exporter is exposing metrics
curl http://<vm-ip>:9100/metrics | head -20

# Check Prometheus health
curl http://<vm-ip>:9090/-/healthy

# Query Prometheus targets via API
curl http://<vm-ip>:9090/api/v1/targets

# View Prometheus logs
sudo docker compose logs prometheus

# View node_exporter logs
sudo docker compose logs node-exporter

# Reload Prometheus config without restart
curl -X POST http://<vm-ip>:9090/-/reload

Logging and Debugging¶

Prometheus Web UI: browse to http://<vm-ip>:9090. The Status → Targets page shows whether scrape targets are UP or DOWN.
PromQL test: in the Graph tab, enter up{} — a value of 1 means the target is being scraped successfully.
Target UNKNOWN state: Prometheus hasn't scraped yet. Wait 15–30 seconds after starting.
connection refused on scrape: the node-exporter service isn't running, or the target name in prometheus.yml is wrong.

Troubleshooting checklist:

sudo docker compose ps — are both containers running?
curl http://localhost:9100/metrics (from the VM) — does node_exporter respond?
Prometheus Targets page — is the node-exporter job UP?
sudo firewall-cmd --list-ports — are 9090 and 9100 open?

Security Considerations¶

No authentication by default: both services expose data without authentication. Restrict access via firewall rules.
Host filesystem access: node_exporter mounts /proc, /sys, and / read-only. Use only the official prom/node-exporter image.
Data persistence: Prometheus data is on a named Docker volume. Deleting the volume loses all historical metrics.