Skip to content

Jaeger

What Is It?

Jaeger is an open-source distributed tracing platform. It collects, stores, and visualizes traces — records of how a request flows through a system, broken down into spans (individual timed operations). Jaeger helps identify slow operations, errors, and dependencies between services.

In this course, a central Jaeger instance is hosted at https://jaeger.sysadm.ee. You configure your inventory API to push traces to it using the OpenTelemetry Protocol (OTLP), and view your traces in the Jaeger web UI.

Key Concepts

Trace
A complete record of a single request as it travels through your application. A trace is made up of one or more spans.
Span
A single named, timed operation within a trace. For example: GET /api/v1/inventory, validate_inventory_cache, db.query.
Service name
How Jaeger identifies which application produced a trace. In this course your service is named inventory.<vm_name>.sysadm.ee.
OTLP
OpenTelemetry Protocol. The standard protocol for sending traces (and metrics/logs) to a backend. Jaeger accepts OTLP over HTTP at /v1/traces.

Default Ports

The central Jaeger instance exposes everything through HTTPS:

URL Purpose
https://jaeger.sysadm.ee/ Web UI (OIDC-protected)
https://jaeger.sysadm.ee/v1/traces OTLP HTTP trace ingestion (no auth required)

Configuration

Instrumenting Python (Flask) with OpenTelemetry

1. Add packages to requirements.txt:

opentelemetry-sdk
opentelemetry-exporter-otlp-proto-http
opentelemetry-instrumentation-flask

2. Add tracing setup to your application (before the first route):

import os
import time
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor

# Read configuration from environment variables
service_name = os.environ.get("OTEL_SERVICE_NAME", "inventory-api")
otlp_endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4318")

# Set up the tracer
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint=f"{otlp_endpoint}/v1/traces")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

# Auto-instrument all Flask routes
FlaskInstrumentor().instrument_app(app)


def validate_inventory_cache():
    """Cache validation step — intentionally slow for tracing exercise."""
    with tracer.start_as_current_span("validate_inventory_cache"):
        time.sleep(0.3)
        return True

Call validate_inventory_cache() from your main inventory route so that every request generates a trace with this span.

3. Configure Docker Compose environment variables (in /home/centos/lab10/docker-compose.yml):

inventory:
  environment:
    - OTEL_EXPORTER_OTLP_ENDPOINT=https://jaeger.sysadm.ee
    - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
    - OTEL_SERVICE_NAME=inventory.<vm_name>.sysadm.ee

Replace <vm_name> with your actual VM short name (e.g. student-xyz).

Important Environment Variables

OTEL_EXPORTER_OTLP_ENDPOINT
The base URL of the OTLP HTTP endpoint. The exporter appends /v1/traces to this URL.
OTEL_EXPORTER_OTLP_PROTOCOL
Must be http/protobuf when using the OTLP HTTP exporter over HTTPS.
OTEL_SERVICE_NAME
The name under which your service appears in the Jaeger UI. Use your VM's FQDN: inventory.<vm_name>.sysadm.ee.

Using the Jaeger UI

  1. Browse to https://jaeger.sysadm.ee and log in with your university account.
  2. In the Search panel, select your service from the Service dropdown.
  3. Click Find Traces — you should see a list of recent traces.
  4. Click on a trace to open the waterfall view showing all spans and their durations.
  5. The span that takes the most time (~300 ms) is the slow one you added. Note its Operation Name.

Common Commands

# Trigger a trace by making a request to the inventory API
curl -H "Authorization: Bearer 845e6732f32b81dd778972703474ccbb" \
  http://inventory.<vm_name>.sysadm.ee/api/v1/inventory

# Query Jaeger HTTP API for your service (from scoring server)
curl "http://localhost:16686/api/services"

# View inventory container logs (check for OTEL errors)
sudo docker compose -f /home/centos/lab10/docker-compose.yml logs inventory

Logging and Debugging

  • Service not appearing in Jaeger: verify OTEL_SERVICE_NAME is set, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf is present, and the inventory container was restarted after the env vars were added.
  • OTLP export errors in container logs: verify OTEL_EXPORTER_OTLP_ENDPOINT=https://jaeger.sysadm.ee and OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf. The exporter needs TLS support, which requires Python's certifi package (usually included in the base image).
  • BatchSpanProcessor drops spans: this can happen if the exporter can't reach Jaeger. Check connectivity from the container: docker exec <container> curl -s https://jaeger.sysadm.ee/v1/traces.
  • Traces appear in Jaeger but validate_inventory_cache span is missing: confirm that validate_inventory_cache() is being called in the request path.

Further Reading