Unit 6.3 · Tutorial 2

Logging & Distributed Tracing

Chapter 19 · Advanced Topics
~2.5 hours Advanced Logging · ELK · Loki · Jaeger · Tracing

Overview

Logging and distributed tracing are essential for understanding complex systems. This tutorial covers structured logging, the ELK stack, Loki, and distributed tracing with Jaeger. You'll learn to implement correlation IDs and trace requests across multiple services.

Why this matters: In microservices and distributed systems, understanding how a request flows through multiple services is critical for debugging and performance analysis.

1. Logging Fundamentals

Logs are the primary source of information for debugging and monitoring.

  • Log levels: DEBUG, INFO, WARN, ERROR, FATAL.
  • Log format: Structured (JSON) vs unstructured (plain text).
  • Best practices: Include timestamp, service name, and correlation ID.
// Unstructured logging (bad) console.log('User 123 logged in successfully'); // Structured logging (good) console.log(JSON.stringify({ level: 'info', timestamp: new Date().toISOString(), service: 'auth-api', userId: 123, event: 'user_login', message: 'User logged in successfully' }));

2. Structured Logging

Structured logs are machine‑readable (JSON) and searchable.

// Node.js with pino import pino from 'pino'; const logger = pino({ level: process.env.LOG_LEVEL || 'info', base: { service: 'my-api', env: process.env.NODE_ENV } }); logger.info({ userId: 123, event: 'user_login' }, 'User logged in'); logger.error({ err: error, userId: 123 }, 'Failed to process request'); // Log output: // {"level":30,"time":1693567890123,"pid":1234,"hostname":"server","service":"my-api","env":"production","userId":123,"event":"user_login","msg":"User logged in"}
// Python with structlog import structlog import logging structlog.configure( processors=[ structlog.processors.TimeStamper(fmt="iso"), structlog.processors.JSONRenderer() ], wrapper_class=structlog.make_filtering_bound_logger(logging.INFO) ) logger = structlog.get_logger() logger.info("user_login", user_id=123, event="User logged in")

3. ELK Stack & Loki

ELK Stack (Elasticsearch, Logstash, Kibana)

  • Elasticsearch: Stores and indexes logs.
  • Logstash: Processes and transforms logs.
  • Kibana: Visualises and searches logs.
# Logstash pipeline configuration input { beats { port => 5044 } } filter { json { source => "message" } date { match => [ "timestamp", "ISO8601" ] } } output { elasticsearch { hosts => ["localhost:9200"] } stdout { codec => rubydebug } }

Loki (Grafana Labs)

Loki is a lightweight log aggregation system that integrates with Grafana.

# docker-compose for Loki services: loki: image: grafana/loki:latest ports: - "3100:3100" command: -config.file=/etc/loki/local-config.yaml promtail: image: grafana/promtail:latest volumes: - /var/log:/var/log command: -config.file=/etc/promtail/config.yml

4. Distributed Tracing (Jaeger)

Distributed tracing tracks a request as it flows through multiple services.

Key concepts

  • Trace: The complete journey of a request.
  • Span: A single operation within a trace (with start/end time).
  • Parent‑child relationships: Spans can have nested relationships.
// Jaeger client (Node.js) import { initTracer, JaegerTracer } from 'jaeger-client'; const config = { serviceName: 'order-service', sampler: { type: 'const', param: 1 }, reporter: { collectorEndpoint: 'http://jaeger:14268/api/traces' } }; const tracer = initTracer(config); // Create a trace const parentSpan = tracer.startSpan('process-order'); parentSpan.setTag('orderId', '12345'); // Create a child span const childSpan = tracer.startSpan('validate-payment', { childOf: parentSpan }); childSpan.setTag('amount', 99.99); childSpan.finish(); parentSpan.finish(); // Or use OpenTelemetry (recommended for newer projects)
// OpenTelemetry with Node.js import { NodeTracerProvider } from '@opentelemetry/node'; import { JaegerExporter } from '@opentelemetry/exporter-jaeger'; import { trace, context } from '@opentelemetry/api'; const provider = new NodeTracerProvider(); const exporter = new JaegerExporter({ endpoint: 'http://jaeger:14268/api/traces' }); provider.addSpanProcessor(new SimpleSpanProcessor(exporter)); provider.register(); const tracer = trace.getTracer('my-service'); const span = tracer.startSpan('handle-request'); const ctx = trace.setSpan(context.active(), span); // ... do work span.end();

5. Correlation IDs

Correlation IDs link logs and traces across services. Every request receives a unique ID that is passed to all downstream services.

// Express middleware for correlation IDs import { v4 as uuidv4 } from 'uuid'; import pino from 'pino'; const logger = pino(); app.use((req, res, next) => { const correlationId = req.headers['x-correlation-id'] || uuidv4(); req.correlationId = correlationId; res.setHeader('x-correlation-id', correlationId); // Attach to logger req.logger = logger.child({ correlationId }); next(); }); // Use in handlers app.get('/api/orders', (req, res) => { req.logger.info({ userId: 123 }, 'Fetching orders'); // ... processing });
// HTTP client that passes correlation ID const axios = require('axios'); async function callDownstream(url, correlationId) { return axios.get(url, { headers: { 'x-correlation-id': correlationId } }); } // Usage in a service const correlationId = req.correlationId; const result = await callDownstream('http://payment-service/api/pay', correlationId);
Best practices:
  • Generate a new correlation ID if none is provided.
  • Forward the ID to all downstream services.
  • Include the ID in all log entries.
  • Return the ID in the response headers.

Quiz

Question 1

What format is recommended for structured logging?

  • CSV
  • JSON
  • XML
  • YAML
Show answer
B. JSON.

Question 2

What is the purpose of a correlation ID?

  • To encrypt requests
  • To link logs and traces across services
  • To authenticate users
  • To cache responses
Show answer
B. To link logs and traces across services.

Question 3

Which tool is used for distributed tracing?

  • Prometheus
  • Jaeger
  • Grafana
  • Loki
Show answer
B. Jaeger.

Exercises

Exercise 1

Implement structured logging with correlation ID in an Express API. Log the correlation ID, request method, and path for every request.

Sample answer
import express from 'express'; import { v4 as uuidv4 } from 'uuid'; import pino from 'pino'; const app = express(); const baseLogger = pino(); app.use((req, res, next) => { const correlationId = req.headers['x-correlation-id'] || uuidv4(); req.correlationId = correlationId; res.setHeader('x-correlation-id', correlationId); req.logger = baseLogger.child({ correlationId }); req.logger.info({ method: req.method, path: req.path }, 'Request received'); next(); }); app.get('/api/users', (req, res) => { req.logger.info({ userId: 123 }, 'Fetching users'); res.json([]); });

Exercise 2

Add OpenTelemetry tracing to a simple microservice. Export traces to Jaeger.

Sample answer
import { NodeTracerProvider } from '@opentelemetry/node'; import { JaegerExporter } from '@opentelemetry/exporter-jaeger'; import { SimpleSpanProcessor } from '@opentelemetry/tracing'; import { trace } from '@opentelemetry/api'; const provider = new NodeTracerProvider(); const exporter = new JaegerExporter({ endpoint: 'http://localhost:14268/api/traces', }); provider.addSpanProcessor(new SimpleSpanProcessor(exporter)); provider.register(); const tracer = trace.getTracer('my-service'); // In your request handler: const span = tracer.startSpan('handle-request'); span.setAttribute('method', 'GET'); // ... process request span.end();

Homework

Homework 1

Implement structured logging and distributed tracing in your capstone project. Add correlation IDs, structured log entries, and trace instrumentation for the critical paths.

Sample outline
  • Logging: Use pino (Node.js) or structlog (Python) with JSON format.
  • Correlation ID: Add middleware to generate/pass correlation IDs.
  • Tracing: Add OpenTelemetry instrumentation for HTTP endpoints.
  • Visualisation: Send logs to Loki and traces to Jaeger.

Mini‑Project

Observability Stack

Build a complete observability stack for your capstone project:

  • Structured logging with correlation IDs
  • Loki (or ELK) for log aggregation
  • Jaeger for distributed tracing
  • Grafana for unified visualisation
  • Combine logs, metrics, and traces
Sample outline
  • Logs: Structured JSON logs from your application.
  • Loki: Log aggregation with Grafana integration.
  • Jaeger: Trace collection and visualisation.
  • Grafana: Unified dashboard with logs, traces, and metrics.
  • Correlation: Link logs to traces using correlation IDs.

Tutorial Summary

You learned to implement structured logging and distributed tracing for complex systems. You used correlation IDs to link logs and traces, set up the ELK stack and Loki for log aggregation, and integrated Jaeger for distributed tracing. These skills are essential for debugging and monitoring modern, distributed applications.

Key takeaway: Observability (logs + metrics + traces) is the foundation of reliable, maintainable systems. Invest in good instrumentation from the start.