Chapter 02 — OpenTelemetry¶

OpenTelemetry is the vendor-neutral, CNCF-graduated standard for collecting, processing, and exporting telemetry data. It is the collection backbone of every production AIOps platform.

OpenTelemetry collection architecture

Poster: thin edge collectors, governed gateways, explicit quality signals, and portable exports.

Prerequisites¶

01 — Observability — must understand metrics, logs, and traces concepts
Basic Kubernetes knowledge (DaemonSet, Deployment, ConfigMap)

03 — Prometheus — receives metrics from OTel Collector
04 — Loki — receives logs from OTel Collector
05 — Tempo — receives traces from OTel Collector
07 — Kafka — OTel Collector can export to Kafka

Next Reading¶

After this chapter, continue to 03 — Prometheus.

1. Why OpenTelemetry?¶

KEY IDEA

Before OpenTelemetry, every observability vendor had its own agent — Datadog agent, New Relic agent, Jaeger client... Changing vendors meant rewriting instrumentation code in every service. OTel solves this with "instrument once, export anywhere": integrate the OTel SDK once, then send data to any backend (Prometheus, Loki, Tempo, Datadog, CloudWatch) by changing collector configuration only.

Tip

Why choose OTel over a vendor agent? Three reasons: (1) No vendor lock-in — if you move from Datadog to Prometheus, only change exporter config, not code. (2) One agent instead of many — OTel Collector handles metrics + logs + traces instead of three separate agents. (3) Open CNCF standard — not dependent on any company's roadmap.

The Problem Before OTel¶

Before OpenTelemetry:

Datadog Agent       → Datadog backend
New Relic Agent     → New Relic backend  
Jaeger Client       → Jaeger backend
→ Change vendor = rewrite instrumentation in all services
→ Run many agents = CPU/memory overhead

What OTel Solves¶

graph LR
    subgraph Before["Before OpenTelemetry"]
        A1[Service] -->|Datadog SDK| DD[Datadog]
        A2[Service] -->|NR SDK| NR[New Relic]
        A3[Service] -->|Zipkin SDK| ZP[Zipkin]
    end

    subgraph After["After OpenTelemetry"]
        B1[Service] -->|OTel SDK| COL[OTel Collector]
        B2[Service] -->|OTel SDK| COL
        B3[Service] -->|OTel SDK| COL
        COL -->|config only| P[Prometheus]
        COL -->|config only| L[Loki]
        COL -->|config only| T[Tempo]
        COL -->|config only| DD2[Datadog]
    end

    style Before fill:#fae8ff,color:#1e293b
    style After fill:#dbeafe,color:#1e293b

Benefits:

Instrument once, export anywhere — change backends without changing application code
Vendor neutral — CNCF graduated, no proprietary lock-in
Unified data model — metrics, logs, traces share the same resource/attribute model
Single agent — OTel Collector replaces multiple vendor agents

OTel vs Other Collection Options¶

Tool	Strengths	Weaknesses	Best for
OTel Collector	Full signals, extensible, vendor-neutral	Complex configuration	Production AIOps (recommended)
Fluent Bit	Extremely light (< 1MB RAM); battle-tested log tail	Logs-first; weaker multi-signal / sampling	Edge / constrained nodes
Vector	Strong transforms (VRL); good routing; multi-sink	Another agent to own; metrics/traces less “OTel-native” than Collector	Log-heavy pipelines, complex routing
Grafana Alloy	Grafana-stack native; replaces Promtail paths; OTel-capable	Best if already Grafana-centric	Prometheus + Loki + Grafana shops
Fluentd	Rich plugin ecosystem	Heavier, Ruby-based	Legacy systems
Prometheus (scrape)	Native Prometheus support	Metrics only, pull-based	Prometheus-native environments
Datadog / vendor agent	Easy setup, full-featured	Vendor lock-in, expensive, parallel agents	Teams standardized on one vendor

Decision: Use OTel Collector as the AIOps gateway. Use Fluent Bit / Vector / Alloy at the edge when you need a thinner or stack-specific shipper — then forward to the Collector (or Kafka), not three parallel brains.

Tip

Do not confuse layers: Fluent Bit / Vector / Alloy / OTel are collect & light process. Flink / Kafka Streams are bus-side stream process after Kafka (06 §5.6). Elasticsearch / Loki are stores. Comparing Fluent Bit to Flink is a category error.

2. OTel Components Overview¶

KEY IDEA

OTel has two separate parts: SDK (in application code — collects telemetry) and Collector (standalone service — processes and routes telemetry). The SDK is like a "sensor," the Collector like a "signal processing station" before storage.

graph TD
    subgraph SDK["OTel SDK (in application)"]
        API[OTel API\nTrace · Metric · Log API]
        IMPL[OTel SDK Implementation\nSpan processor\nMetric reader]
        PROP[Context Propagator\nW3C TraceContext\nB3 Headers]
        RES[Resource\nservice.name, version\nhost, k8s attributes]
    end

    subgraph Collector["OTel Collector"]
        RCV[Receivers\nOTLP · Prometheus · Jaeger\nZipkin · Kafka]
        PROC[Processors\nBatch · MemoryLimiter\nTransform · Filter\nTailSampling · Attributes]
        EXP[Exporters\nOTLP · Prometheus\nLoki · Kafka · Datadog]
    end

    subgraph Backends["Storage Backends"]
        PM[Prometheus]
        LK[Loki]
        TP[Tempo]
        KF[Kafka]
    end

    SDK -->|OTLP gRPC :4317\nOTLP HTTP :4318| Collector
    Collector --> Backends

    style SDK fill:#dbeafe,color:#1e293b
    style Collector fill:#dcfce7,color:#1e293b
    style Backends fill:#f3e8ff,color:#1e293b

3. OTLP Protocol¶

KEY IDEA

OTLP (OpenTelemetry Protocol) is the "language" SDKs and Collectors speak. There are 3 variants, differing in format and performance. In production, use gRPC (binary, fastest). Use HTTP/JSON only for debug or browser clients.

Protocol Variants¶

Variant	Port	Format	When to use
OTLP gRPC	4317	Protobuf binary	Default for service→collector. Most efficient.
OTLP HTTP/protobuf	4318	Protobuf binary	When gRPC is not possible
OTLP HTTP/JSON	4318	JSON	Debug, browser apps

OTLP Data Model — Trace (simplified)¶

{
  "resourceSpans": [{
    "resource": {
      "attributes": [
        {"key": "service.name", "value": {"stringValue": "order-service"}},
        {"key": "k8s.pod.name", "value": {"stringValue": "order-svc-abc123"}}
      ]
    },
    "scopeSpans": [{
      "spans": [{
        "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
        "spanId": "00f067aa0ba902b7",
        "name": "POST /api/orders",
        "startTimeUnixNano": 1705329825050000000,
        "endTimeUnixNano": 1705329825115000000,
        "status": {"code": "STATUS_CODE_OK"}
      }]
    }]
  }]
}

4. The OTel Collector Deep Dive¶

KEY IDEA

The Collector is a 3-stage pipeline: Receivers (ingest data), Processors (transform/filter/sample), Exporters (send to backends). You can configure multiple parallel pipelines — one for traces, one for metrics, one for logs — each with its own processors and exporters.

Internal Architecture¶

graph LR
    subgraph Receiver["Receivers"]
        R1[OTLP\ngRPC :4317\nHTTP :4318]
        R2[Prometheus\nscrape /metrics]
        R3[Kafka Receiver]
    end

    subgraph Pipeline["Internal Pipeline (order matters!)"]
        P1[Memory Limiter\n← ALWAYS FIRST]
        P2[Filter\ndrop health checks]
        P3[Transform\nenrich metadata]
        P4[Tail Sampling\nkeep errors/slow]
        P5[Batch\nbatch for export]
    end

    subgraph Exporter["Exporters"]
        E1[OTLP → Tempo]
        E2[Prometheus RW]
        E3[Loki]
        E4[Kafka → AIOps]
    end

    Receiver --> P1 --> P2 --> P3 --> P4 --> P5 --> Exporter

    style Receiver fill:#dbeafe,color:#1e293b
    style Pipeline fill:#dcfce7,color:#1e293b
    style Exporter fill:#f3e8ff,color:#1e293b

Collector Distributions¶

Distribution	Description	When to use
otelcol	Core, minimum components	Extremely small resources
otelcol-contrib	Full community components	Most common for production
Custom build (ocb)	Only components you need	High-security production

5. Receiver Configuration¶

KEY IDEA

Receivers are the Collector's "ingress." Most important are the OTLP Receiver (from services) and the Prometheus Receiver (pull-based scraping). OTLP gRPC is the default and most efficient.

OTLP Receiver¶

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 4         # Max message size
        max_concurrent_streams: 1000      # Concurrent gRPC streams
        tls:
          cert_file: /certs/server.crt   # mTLS — required for production
          key_file: /certs/server.key
          client_ca_file: /certs/ca.crt

      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins: ["https://your-frontend.com"]  # Browser clients

Prometheus Receiver (pull-based)¶

receivers:
  prometheus:
    config:
      global:
        scrape_interval: 15s
      scrape_configs:
        - job_name: k8s-pods
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            # Only scrape pods with annotation prometheus.io/scrape: "true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: "true"

Kafka Receiver¶

receivers:
  kafka:
    brokers: ["kafka-1:9092", "kafka-2:9092"]
    topic: otlp-telemetry
    group_id: otel-collector-consumer
    encoding: otlp_proto
    auth:
      sasl:
        username: otel-collector
        password: ${KAFKA_PASSWORD}
        mechanism: SCRAM-SHA-512

6. Processor Configuration¶

KEY IDEA

Processors are the most important part of the Collector — they determine data quality, resource use, and cost. Processor order matters:

memory_limiter → filter → transform → tail_sampling → batch

Violating this order can cause data loss or collector crashes.

Tip

Why does processor order matter? If you put batch before tail_sampling, spans of the same trace are batched into different batches → the tail sampler cannot decide correctly for the whole trace. Without memory_limiter first → collector OOM crashes on traffic spikes.

Memory Limiter Processor (ALWAYS FIRST)¶

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 3000          # Hard limit: refuse new data when exceeded
    spike_limit_mib: 500     # Buffer for traffic spikes

# At 3500 MiB (3000+500), collector starts refusing new spans
# This backpressure mechanism prevents OOM crash — more important than losing a little data

Batch Processor¶

processors:
  batch:
    send_batch_size: 8192       # Send when this size is reached
    send_batch_max_size: 16384  # Max size per batch
    timeout: 5s                 # Send after this time even if size not reached

# Why batch matters: Reduces RPC calls 10-100×
# 8192 spans × 1KB = 8MB/batch → Tempo is not overloaded by tiny exports

Transform Processor¶

Tip

Why is Transform the "smartest" processor? It lets you enrich data (add environment from k8s namespace), normalize (lowercase service name), scrub sensitive data (replace SQL values with ?), and extract structured fields from unstructured text — all without changing application code.

processors:
  transform/enrich:
    error_mode: ignore         # Do not drop data if transform fails

    trace_statements:
      - context: resource
        statements:
          # Attach environment from k8s namespace
          - set(attributes["deployment.environment"], "production") where IsMatch(attributes["k8s.namespace.name"], "prod.*")

      - context: span
        statements:
          # Scrub SQL values to avoid storing sensitive data
          - replace_pattern(attributes["db.statement"], "'[^']*'", "?")
          - replace_pattern(attributes["db.statement"], "\\d+", "?")

    log_statements:
      - context: log
        statements:
          # Normalize severity for legacy logs
          - set(severity_number, SEVERITY_NUMBER_ERROR) where attributes["level"] == "FATAL"

Filter Processor¶

processors:
  filter/drop_noise:
    traces:
      span:
        # Drop health check endpoints — no value for AIOps
        - 'attributes["http.route"] == "/health"'
        - 'attributes["http.route"] == "/ready"'
        - 'IsMatch(attributes["http.user_agent"], "kube-probe.*")'

    metrics:
      metric:
        # Drop Go runtime metrics — high cardinality, low AIOps value
        - 'IsMatch(name, "go_gc_.*")'

    logs:
      log_record:
        # Drop DEBUG/TRACE in production — reduce 80-90% log volume
        - 'severity_number < SEVERITY_NUMBER_WARN'

Tail Sampling Processor¶

KEY IDEA

Tail sampling is the "smart filter" for traces — wait until the trace completes before deciding keep or drop. Result: 100% of errors kept, 5-10% of normal traffic kept. Compared with head sampling (decide before knowing the outcome), tail sampling is better because it always keeps the most important traces.

Memory trade-off: Must hold all spans in memory for decision_wait seconds. 50K traces × 50 spans × 2KB ≈ 5GB RAM. That is why the gateway needs 4-8GB RAM.

processors:
  tail_sampling:
    decision_wait: 30s          # Wait up to 30s to collect enough spans
    num_traces: 50000           # Max 50K traces in memory
    expected_new_traces_per_sec: 500

    policies:
      # Always keep traces with errors — highest priority
      - name: sample-errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Always keep slow traces (> 2 seconds) — next priority
      - name: sample-slow
        type: latency
        latency:
          threshold_ms: 2000

      # Always keep payment traces — business critical
      - name: sample-payment
        type: string_attribute
        string_attribute:
          key: service.name
          values: [payment-service, billing-service]

      # 5% of normal traffic — statistical baseline
      - name: sample-normal-5pct
        type: and
        and:
          and_sub_policy:
            - name: not-error
              type: status_code
              status_code:
                status_codes: [OK, UNSET]
            - name: probabilistic
              type: probabilistic
              probabilistic:
                sampling_percentage: 5

Attributes Processor¶

processors:
  attributes/add_metadata:
    actions:
      - key: collector.version   # Add version info for debug
        value: "0.95.0"
        action: insert

      - key: user.id             # Hash sensitive IDs — do not delete, still debuggable
        action: hash

      - key: http.request.header.authorization  # Fully delete auth headers
        action: delete

7. Exporter Configuration¶

KEY IDEA

Exporters are the "egress" — send processed data to backends. Each backend has its exporter: OTLP→Tempo, Prometheus Remote Write→Prometheus, Loki Exporter→Loki, Kafka→AIOps pipeline. All exporters need retry logic and queues to avoid data loss when backends are temporarily unavailable.

OTLP Exporter (→ Tempo for traces)¶

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.observability.svc.cluster.local:4317
    tls:
      ca_file: /certs/ca.crt
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s        # Retry for at most 5 minutes
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 1000
      storage: file_storage/traces  # Persistent queue — no data loss on crash
    compression: gzip

Prometheus Remote Write Exporter (→ Prometheus)¶

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus.observability.svc.cluster.local:9090/api/v1/write
    resource_to_telemetry_conversion:
      enabled: true   # Convert resource attributes into metric labels
    retry_on_failure:
      enabled: true

Loki Exporter (→ Loki for logs)¶

exporters:
  loki:
    endpoint: http://loki-distributor.observability.svc.cluster.local:3100/loki/api/v1/push
    default_labels_enabled:
      level: true     # Use severity as label — low cardinality, high value
    retry_on_failure:
      enabled: true

Kafka Exporter (→ AIOps pipeline)¶

exporters:
  kafka/aiops:
    brokers: ["kafka-1.kafka.svc:9092", "kafka-2.kafka.svc:9092"]
    topic: aiops-raw-telemetry
    encoding: otlp_proto
    producer:
      required_acks: 1              # Leader ack enough — trade latency vs durability
      compression: snappy
    auth:
      sasl:
        username: ${KAFKA_USER}
        password: ${KAFKA_PASSWORD}
        mechanism: SCRAM-SHA-512

File Storage Extension (persistent queue)¶

extensions:
  file_storage/traces:
    directory: /var/lib/otelcol/storage/traces
    compaction:
      on_start: true
      rebound_needed_threshold_mib: 100

8. Pipeline Definition¶

ILLUSTRATION — Complete pipeline configuration

This is the skeleton config combining all components above into a complete pipeline. Note: each signal type (traces/metrics/logs) has its own pipeline with different processors and exporters.

service:
  extensions: [health_check, pprof, zpages, file_storage/traces]

  pipelines:
    # Traces: receive → filter noise → enrich → tail sample → batch → export
    traces:
      receivers: [otlp, jaeger, zipkin]
      processors:
        - memory_limiter          # ALWAYS first — OOM circuit breaker
        - filter/drop_noise       # Drop health checks, k8s probes
        - transform/enrich        # Add environment, clean SQL
        - attributes/add_metadata # Attach collector metadata
        - tail_sampling           # Keep errors + slow + 5% normal
        - batch                   # Batch AFTER sampling — correct order!
      exporters: [otlp/tempo, kafka/aiops]

    # Metrics: receive → filter → enrich → batch → Prometheus
    metrics:
      receivers: [otlp, prometheus]
      processors:
        - memory_limiter
        - filter/drop_noise
        - transform/enrich
        - batch
      exporters: [prometheusremotewrite]

    # Logs: receive → filter → mask PII → enrich → batch → Loki + Kafka
    logs:
      receivers: [otlp]
      processors:
        - memory_limiter
        - filter/drop_noise       # Drop DEBUG/TRACE
        - transform/mask_pii      # Hide PII before storage
        - transform/enrich
        - batch
      exporters: [loki, kafka/aiops]

  # Collector self-monitoring
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8888      # Prometheus scrapes metrics here

9. Deployment Patterns¶

Pattern 1: Agent + Gateway (Recommended for Production)¶

KEY IDEA

Use 2 tiers: Agent (DaemonSet on each node, light, only forwards) + Gateway (Deployment with multiple replicas, does heavy processing like tail sampling, batching). Why 2 tiers? Because tail sampling needs to see all spans of a trace — if services A and B run on different nodes, their spans come from different agents → must assemble at the Gateway.

graph LR
    subgraph Node1["K8s Node 1"]
        P1[Pod A] -->|OTLP| AG1[OTel Agent\nDaemonSet]
        P2[Pod B] -->|OTLP| AG1
        AG1 -->|OTLP| GW
    end

    subgraph Node2["K8s Node 2"]
        P3[Pod C] -->|OTLP| AG2[OTel Agent\nDaemonSet]
        AG2 -->|OTLP| GW
    end

    subgraph Gateway["Gateway (×3 replicas)"]
        GW[OTel Collector Gateway\nTail Sampling + Routing]
    end

    GW -->|remote_write| PROM[Prometheus]
    GW -->|push| LOKI[Loki]
    GW -->|OTLP| TEMPO[Tempo]
    GW -->|produce| KAFKA[Kafka]

Aspect	Agent (DaemonSet)	Gateway (Deployment)
CPU/RAM	200m CPU, 256Mi	2 CPU, 4Gi
Tail sampling	❌ Cannot (spans distributed)	✅ Can assemble
HA	Natural (per node)	Need ≥3 replicas

Agent configuration (light — forward only, no sampling):

# otel-agent-config.yaml (DaemonSet)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    limit_mib: 200            # Small limit for agent
    spike_limit_mib: 50
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  otlp/gateway:
    endpoint: otel-collector-gateway.observability.svc.cluster.local:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]   # No tail_sampling!
      exporters: [otlp/gateway]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/gateway]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/gateway]

Pattern 2: Sidecar (For specialized services)¶

Use when a service needs special processing (e.g. 100% sampling for payment service):

spec:
  containers:
    - name: payment-service
      env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://localhost:4317"  # Send to sidecar

    - name: otel-collector         # Dedicated sidecar collector
      image: otelcol-contrib:0.95.0
      resources:
        requests: { cpu: "100m", memory: "128Mi" }
        limits:   { cpu: "500m", memory: "512Mi" }

10. Kubernetes Operator¶

KEY IDEA

The OTel Operator is a Kubernetes controller — it manages OTel Collector lifecycle and auto-injects instrumentation into pods just by adding annotations. No container image code changes required. This is "zero-code instrumentation" for entire namespaces.

Installing the Operator¶

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

OpenTelemetryCollector CRD¶

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: aiops-collector
  namespace: observability
spec:
  mode: daemonset
  image: otelcol-contrib:0.95.0
  resources:
    limits:
      cpu: "500m"
      memory: "512Mi"
    requests:
      cpu: "200m"
      memory: "256Mi"
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
    # ... full collector config

Auto-Instrumentation CRD¶

Tip

Why is Auto-Instrumentation a "killer feature"? Instead of requiring every team to add the OTel SDK to their code, you annotate the namespace → Operator injects agents into every new pod. Teams change nothing in code, yet get full traces/metrics.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: aiops-instrumentation
  namespace: production
spec:
  exporter:
    endpoint: http://aiops-collector.observability.svc.cluster.local:4317

  propagators:
    - tracecontext
    - baggage

  sampler:
    type: parentbased_traceidratio
    argument: "0.1"         # SDK-level 10% sampling (before tail sampling)

  java:   # Auto-inject Java agent
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.32.0
  python: # Auto-inject Python agent
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.43b0
  nodejs: # Auto-inject Node.js agent
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.45.0

Enable auto-instrumentation for an entire namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    instrumentation.opentelemetry.io/inject-python: "true"
    # Every new pod in this namespace is auto-instrumented

11. Fluent Bit vs OTel Collector (and Vector / Alloy)¶

KEY IDEA

Edge shippers are not rivals to each other in the abstract — they trade RAM, signal coverage, and transform power. Fluent Bit excels at log collection with extremely low resources. OTel Collector is the choice when you need multi-signal processing and advanced features like tail-based sampling. Vector and Grafana Alloy sit between “log router” and “OTel-compatible agent” depending on how you configure them.

Criterion	Fluent Bit	Vector	Grafana Alloy	OTel Collector
RAM footprint	~1MB class	Higher than FB	Medium	256MB+ typical gateway
Signals	Logs-first	Logs + metrics (varies)	Metrics/logs/traces (Grafana + OTel paths)	Metrics + logs + traces
Tail sampling	❌	❌ / limited	Via OTel-style pipelines	✅
Transforms	Basic filters	Strong (VRL)	Good (River / components)	Processors + OTTL
Best AIOps role	Node log edge	Complex log fan-out	Grafana-stack edge	Policy gateway
Production maturity	Extremely high	High	High (Grafana ecosystem)	High (CNCF graduated)

Decision Matrix¶

Need traces with tail sampling / org-wide policy?  → OTel Collector (gateway)
Logs only, constrained resources?                 → Fluent Bit
Complex log route/transform, multi-sink?          → Vector (or Collector)
Already all-in Grafana (Prom/Loki/Tempo)?         → Alloy + OTel gateway optional
Full telemetry platform for AIOps?                → OTel Collector
Legacy Fluentd estate?                            → Fluent Bit edge → migrate to OTel

Pros / cons (edge tools vs each other)¶

Tool	Pros	Cons
Fluent Bit	Tiny; proven DaemonSet; simple	Weak multi-signal story; limited deep policy
Vector	Excellent routing/transforms; observability of the agent	Extra skill (VRL); can duplicate Collector if both grow brains
Alloy	One agent for many Grafana receivers; modern Promtail replacement	Gravity toward Grafana backends
OTel Collector	One policy plane; vendor-neutral export; sampling	Heavier; config complexity

Hybrid Pattern (recommended)¶

Fluent Bit or Vector or Alloy (DaemonSet) → node/system logs
OTel Agent / SDK → application OTLP
        ↘
          OTel Collector Gateway → Prometheus / Loki / Tempo / Kafka

Warning

Running Datadog Agent + OTel + Fluent Bit + Vector on every node “just in case” burns CPU and multiplies cardinality bugs. Pick one edge log shipper + one OTel path, then converge.

12. Production Best Practices¶

Resource Limits (Production Sizing)¶

# Agent (DaemonSet) — light
resources:
  requests: { cpu: "200m", memory: "256Mi" }
  limits:   { cpu: "500m", memory: "512Mi" }

# Gateway (Deployment, with tail sampling) — needs more RAM
resources:
  requests: { cpu: "2000m", memory: "4Gi" }  # 4Gi for tail sampling buffer
  limits:   { cpu: "4000m", memory: "8Gi" }

HorizontalPodAutoscaler for Gateway¶

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: otel-collector-gateway-hpa
spec:
  scaleTargetRef:
    name: otel-collector-gateway
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          averageUtilization: 70
    - type: Pods
      pods:
        metric:
          name: otelcol_receiver_accepted_spans
        target:
          averageValue: "50000"    # Scale up if >50K spans/s per pod

13. Common Mistakes¶

Common mistake	Symptom	Fix
memory_limiter not first	Collector OOM crash on traffic spikes	Always put memory_limiter first
Batch before tail_sampling	Wrong sampling decisions	Tail sample → BATCH (correct order)
No persistent queue	Data loss on restart	Enable file_storage extension
No max_recv_msg_size config	"message too large" errors	Set appropriate `max_recv_msg_size_mib`
Tail sampling at agent	Cannot correlate spans across nodes	Only tail sample at gateway
Auto-instrument everything	500MB JVM agent overhead	Selectively instrument packages
Single collector pod	SPOF	Minimum 3 gateway replicas
Missing trace_id in logs	Cannot navigate log→trace	Mandate trace context injection at SDK

14. Monitoring the Collector¶

KEY IDEA

The Collector exposes Prometheus metrics at :8888/metrics. Most important to watch: refused/failed spans (must be 0 in steady state) and queue size (if steadily rising → collector is overloaded).

# Spans received — baseline throughput
rate(otelcol_receiver_accepted_spans[5m])

# Spans dropped — MUST be 0. If > 0 → collector is overloaded
rate(otelcol_receiver_refused_spans[5m])
rate(otelcol_exporter_failed_spans[5m])

# Queue size — if rising → backend cannot keep up
otelcol_exporter_queue_size / otelcol_exporter_queue_capacity

# Memory — confirm memory_limiter is working
otelcol_process_memory_rss

# Tail sampling decisions — what % is kept
rate(otelcol_processor_tail_sampling_sampled_spans[5m])
rate(otelcol_processor_tail_sampling_not_sampled_spans[5m])

Alerting Rules¶

groups:
  - name: otel-collector
    rules:
      - alert: OTelCollectorHighDropRate
        expr: |
          rate(otelcol_exporter_failed_spans[5m]) /
          rate(otelcol_receiver_accepted_spans[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "OTel Collector dropping >1% of spans"

      - alert: OTelCollectorQueueFull
        expr: otelcol_exporter_queue_size / otelcol_exporter_queue_capacity > 0.8
        for: 5m
        labels:
          severity: warning

15. Scaling¶

Scaling Bottlenecks¶

Bottleneck	Symptom	Fix
CPU	`otelcol_process_cpu_seconds` high	Add replicas
Memory	OOM kills	Raise memory limit or lower `num_traces`
Network	Queue rises, export retries	Scale Tempo/Loki/Prometheus
gRPC connections	Connection refused from agents	Raise `max_concurrent_streams`

Target Allocator (Prometheus scraping at scale)¶

When one Prometheus instance cannot scrape enough targets, Target Allocator distributes scrape jobs evenly:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
spec:
  mode: statefulset
  replicas: 5
  targetAllocator:
    enabled: true
    allocationStrategy: consistent-hashing    # Stable distribution across restarts
    prometheusCR:
      enabled: true   # Auto-discover ServiceMonitor and PodMonitor CRDs

16. Security¶

mTLS with cert-manager¶

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: otel-collector-cert
  namespace: observability
spec:
  secretName: otel-collector-tls
  duration: 2160h        # 90 days
  renewBefore: 360h      # Auto-renew 15 days before expiry
  usages:
    - server auth
    - client auth
  dnsNames:
    - otel-collector-gateway.observability.svc.cluster.local

Secrets Management¶

# Never hard-code credentials in ConfigMap
# Use external-secrets-operator → AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
spec:
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  data:
    - secretKey: KAFKA_PASSWORD
      remoteRef:
        key: /aiops/otel-collector/kafka-password

17. Cost¶

KEY IDEA

OTel Collector resource cost itself is modest (~$260/month for 10 nodes). The real value is reducing downstream cost through smart sampling and filtering.

OTel Collector Resource Cost¶

Deployment	Instances	Total
DaemonSet Agent (10 nodes)	10 × 0.2 CPU = 2 CPU	~$60/month
Gateway (3 replicas)	3 × 2 CPU = 6 CPU	~$200/month
Total		~$260/month

Data Volume Impact (Cost Savings)¶

Traces — impact of tail sampling:
  No sampling: 1M spans/minute × 2KB = 2.88TB/day
  10% tail sampling: 288GB/day
  Savings: ~$150/day on Tempo S3 storage alone

Logs — impact of filtering:
  No filter: 144GB/day
  Drop DEBUG + sample INFO 10%: 15GB/day
  Savings: ~$6/day for Loki S3

18. Production Problem-Solving Mindset¶

KEY IDEA

OpenTelemetry is not "install the agent and you are done." Production OTel is about context correctness, processing order, and single points of failure (SPOF) on the telemetry path. Strong OTel on-call asks: is the data present, is it correct, and is it cheap enough to survive peak traffic?

18.1 Auto vs Manual Instrumentation — Choose by Questions, Not Fashion¶

Question	Auto-instrumentation	Manual instrumentation
Common framework/HTTP/DB?	Prefer auto	—
Business span (checkout, risk score)?	Not enough	Manual required
Legacy lib without plugin?	May miss	Manual / wrapper
Time-to-value	Fast	Slower, deeper
Overhead / version pin risk	Agent version risk	Code ownership

Why

Auto gives coverage; manual gives meaning. AIOps and RCA need meaning (stable span names, semantic attributes). Winning strategy: auto as foundation + manual for critical paths + semconv.

Rule of thumb:

Tier-1 HTTP/gRPC/DB/messaging → auto first.
Every domain boundary (payment, authz decision, saga step) → manual span.
Do not manual-span every internal function — noise + cost.
Attributes follow semantic conventions; avoid inventing my_company_foo if a standard exists.

18.2 Context Propagation Breaks — "Truncated Trace" Is a Production Bug¶

Context propagation is the backbone of distributed tracing. Breaks are often silent:

Service A creates a root; Service B creates a new root (parent lost).
Async: publish does not inject; consumer does not extract.
Gateway/mesh strips headers (traceparent, baggage).
Language boundary: Java → Node over HTTP missing propagator config.
Fan-out: only the first child has context.

Debug mindset:

1. Pick one concrete request_id/user action
2. Find the first span (root) — which service?
3. Compare network hops vs span tree
4. Which hop has a request but no child span? → break there
5. Check: propagator, header allowlist, middleware order, async context

Edge

"Having traces" ≠ "correct propagation." A colorful Tempo dashboard can still be a forest of disconnected root spans, not one journey. Measure with % of multi-service traces with depth ≥ N.

Two common deployment layers:

Agent (DaemonSet/sidecar): close to the app; fail → that node/pod loses local telemetry.
Gateway (Deployment): tail sampling, routing, fan-out; fail → lose centralized enrichment/sampling policy.

Reliability mindset:

Component	Failure mode	Impact	Mitigation
Agent	OOM / crash	Lose node signals	correct resource limits, memory_limiter, restart policy
Gateway	1 replica	SPOF sampling/export	≥3 replicas + PDB
Gateway LB	round-robin spoils tail sample	Wrong decision	hash by traceID
Exporter backend	Tempo/Loki down	Backpressure / drop	queue + retry + disk WAL
Config reload	bad YAML	Pipeline stops	validation CI, canary collector

18.4 Processor Ordering — Why Order Matters More Than "Having Processors"¶

OTel pipelines are not commutative. Classic wrong examples:

WRONG:  filter → batch → memory_limiter
      (already spent RAM before limiting)

WRONG:  tail_sampling → attributes(add tenant)
      (sample before enough attrs for policy)

CORRECT (simplified):
  memory_limiter → [attributes/filter/transform] → batch → exporters
  (tail_sampling usually near the end of the traces path, after enough enrichment)

Why

memory_limiter first protects the process. batch near exporter optimizes network. filter early reduces CPU. transform before export so backends receive a clean schema. Wrong order = OOM + random drops + wrong sampling policy.

18.5 Problem-Solving Loop for "No Data Seen"¶

App SDK → Agent → Gateway → Backend (Prom/Loki/Tempo)
   |         |        |            |
   export    receive  process      ingest
   errors    refused  drops        rejections

Always go right to left if the backend is empty, or left to right if the app logs export errors:

Is the backend receiving a different tenant/namespace?
Gateway exporter metrics: sent / failed / queue.
Agent → gateway connectivity / mTLS / auth.
SDK endpoint env (OTEL_EXPORTER_OTLP_ENDPOINT) correct port/protocol.
Sampling: may be dropping 99% — use a forced error trace test.

19. Real-World Edge Cases¶

EC-01 — Auto-instrument "has spans" but lacks business context¶


Symptom	Trace only shows `HTTP GET` / `SELECT`; cannot tell which checkout step failed.
Cause	Auto only; no domain manual spans.
Detection	Review span names on critical path; product asks "where?"
Prevention	Instrumentation standard: required business spans list per service.

EC-02 — Manual spans too fine-grained → cost + noise¶


Symptom	500+ spans/request; Tempo slow; bill rises.
Cause	Span every helper; loops create spans.
Detection	p99 spans per trace; collector throughput.
Prevention	Span budget/request; suppress library spans; sample internals.

EC-03 — `traceparent` stripped by API gateway¶


Symptom	Each service is its own root trace; cross-service correlation broken.
Cause	Header allowlist does not include W3C trace context.
Detection	Compare ingress request headers vs app; Tempo service graph broken.
Prevention	Gateway config allow `traceparent`, `tracestate`, baggage; contract test.

EC-04 — Async messaging loses context¶


Symptom	HTTP path has a trace; Kafka consumer is a separate root.
Cause	No inject into message header/carrier.
Detection	Trace tree stops at produce; consumer has no parent.
Prevention	Propagator for Kafka/SQS; e2e produce→consume test.

EC-05 — Single-replica collector gateway¶


Symptom	Gateway deploy → 2–5 minute cluster-wide telemetry gap.
Cause	1 pod; rolling update without PDB.
Detection	Gateway `up` metric; export lag.
Prevention	min 3 replicas, PDB, maxUnavailable 1; pre-stop drain.

EC-06 — Wrong tail sampling when scaling out¶


Symptom	Error traces "disappear" despite 100% error policy.
Cause	Spans of same trace hit different replicas; decisions inconsistent.
Detection	Compare error metrics vs kept error traces; load balancer mode.
Prevention	Consistent hash routing by traceID; or load-balancing exporter.

EC-07 — memory_limiter not first in pipeline¶


Symptom	Collector OOMKill even with limiter set.
Cause	Heavy processors run first; batch inflates RAM.
Detection	Pipeline config review; container memory working set.
Prevention	Lint config: memory_limiter first; load test.

EC-08 — Filter after batch / wrong condition → swallows error logs¶


Symptom	Production missing ERROR logs; only INFO remains.
Cause	Wrong severity filter; transform field name mismatch.
Detection	Canary ERROR log with marker; log count metrics by severity.
Prevention	Unit test OTTL; staged filter rollout; never blindly drop unknown severity.

EC-09 — Java agent version pin causes deadlock / permgen issues¶


Symptom	After agent upgrade, latency/thread spike.
Cause	Incompatible bytecode instrumentation.
Detection	Diff agent version canary; flight recorder.
Prevention	Pin version; 5% canary; rollback playbook; JDK matrix test.

EC-10 — Fan-out exporter: one slow backend backpressures the whole pipeline¶


Symptom	Slow Loki → traces also drop.
Cause	Shared pipeline/queue; blocking export.
Detection	Exporter queue size; failed per backend.
Prevention	Split pipelines; sending_queue; timeout; isolate slow sink.

EC-11 — Missing resource attributes `service.name`¶


Symptom	Grafana shows `unknown_service`; hard to filter.
Cause	SDK resource detector miss; env not set.
Detection	Count spans without service.name.
Prevention	Enforce at collector (transform default); platform chart defaults.

EC-12 — PII leaks into baggage / attributes¶


Symptom	Email/token appears in Tempo/Loki.
Cause	Dev set debug attributes; baggage propagates secrets.
Detection	Attribute scan; DLP sample; security review.
Prevention	Redact processor; attribute allowlist; ban sensitive keys in CI.

20. Decision Trees¶

20.1 Auto, Manual, or Hybrid?¶

flowchart TD
    A[New / legacy service] --> B{OTel auto for stack?}
    B -->|No| C[Manual + library wrappers]
    B -->|Yes| D[Enable auto for HTTP/DB/RPC]
    D --> E{Critical business path?}
    E -->|Yes| F[Add manual spans + semconv attrs]
    E -->|No| G[Keep auto + standard resource attrs]
    F --> H[Set span budget + review attr cardinality]
    C --> H
    G --> H

20.2 Truncated Trace — Narrow the Break¶

flowchart TD
    A[Trace depth = 1 despite multi-service call] --> B{traceparent header reaches service B?}
    B -->|No| C[Gateway/mesh/proxy strip or client not inject]
    B -->|Yes| D{SDK B extract + correct propagator?}
    D -->|No| E[Fix propagator / middleware order]
    D -->|Yes| F{Async / thread hop?}
    F -->|Yes| G[Context lost in executor — wrap context]
    F -->|No| H[Sampling drop child? check collector policy]

20.3 Collector Dropping Data¶

flowchart TD
    A[Missing telemetry] --> B{SDK export errors?}
    B -->|Yes| C[Endpoint / TLS / auth / backpressure]
    B -->|No| D{Agent receive increasing?}
    D -->|No| E[Network path / DNS / NetworkPolicy]
    D -->|Yes| F{Processor drop metrics rising?}
    F -->|Yes| G[Filter/tail_sample/memory_limiter]
    F -->|No| H{Exporter failed?}
    H -->|Yes| I[Backend health + queue + retry]
    H -->|No| J[Wrong tenant/labels — data elsewhere]

20.4 Fixing Processor Order¶

Always ask:
1. What protects the process? → memory_limiter early
2. What reduces volume early? → filter early (after limit)
3. What needs the full signal? → tail_sampling after enrich
4. What optimizes I/O? → batch near exporter

21. Lessons from Big Tech / Public Incidents¶

Why

Vendor-neutral does not immunize against operational failure. Map to Ch13 Big Tech / Ch15 Famous Incidents.

21.1 Agent/sidecar blast radius¶

Large orgs learn that instrumentation agents are runtime dependencies: CVE, perf regression, JDK incompatibility. Pattern: progressive delivery for agents like apps; kill switch to disable instrumentation.

21.2 Sampling policy incidents¶

Public write-ups on tracing cost: low head sample → miss rare bugs; high sample → bill burns. Tail-based + error-aware is the engineering answer, but routing must be correct (traceID affinity).

21.3 Single pipeline / multi-sink coupling¶

When one sink (logs) is slow and pulls metrics/traces down, multi-signal platforms "DDoS themselves." Lesson: isolate pipelines, backpressure per signal, priority queues (metrics > traces > debug logs in crisis).

21.4 Semantic convention drift¶

Without span/attr standards → AIOps and dashboards break after 6 months. Big tech invests in schema governance; platform teams review semconv like API review.

21.5 Mental Map¶

Lesson	OTel chapter	Next
Progressive agent rollout	Operator/auto-instr	Ch12 production
Sampling + cost	Tail sampling processor	Ch01 cost, Ch05 Tempo
Pipeline isolation	Exporters/pipelines	Ch06 Kafka buffer patterns
Schema governance	Attributes/transform	Ch09 RCA features, Ch10 agents

22. Socratic Questions for On-Call¶

22.1 When "There Is No Trace"¶

Are you searching by TraceID, service name, or time range? Which is more trustworthy right now?
Which services did the real request pass through? Evidence from metrics/logs?
Is this no export, filtered, or sampled?
Which collector (agent/gateway) is on the path? What do collector metrics say?
Does the OTLP HTTP/gRPC protocol match the endpoint?

22.2 When the Trace "Exists but Is Useless"¶

Do span names help you tell the business story?
Which attributes are missing to distinguish tenant/version/region?
How many spans/request? Are you creating your own noise?
Are errors marked status=ERROR correctly?
Does parent-child reflect the real dependency graph?

22.3 When Collector/Perf¶

Is memory_limiter first? Who last reviewed order?
If the gateway loses 1 AZ, are sampling decisions still correct?
How does a slow backend affect other signals?
How much CPU/RAM is the auto agent adding vs budget?
Is there PII on attributes — how do you check?

22.4 After the Incident¶

Need more manual spans or less auto noise?
Which test will fail CI if propagation breaks again?
Does the collector OOM runbook have the first 5 commands?
Which 30-day experiment reduces time-to-first-useful-trace?
Is OTel data ready for correlation Ch01 / RCA Ch09?

23. Improvement Experiments (30/60/90 Days)¶

30 Days — Visibility & Safety¶

Experiment	How	Success metric
Propagation audit	10 critical e2e flows	≥80% multi-service traces depth OK
Collector golden signals	Dashboard sent/refused/queue/OOM	Alert on drop rate
Processor order lint	Policy + PR template	100% pipelines memory_limiter first
service.name coverage	Transform default	0 unknown_service tier-1
Agent canary process	1 namespace pin version	Rollback <15 minutes documented

Deliverables: agent→gateway→backends topology diagram; break-point list.

60 Days — Quality & Cost¶

Experiment	How	Success metric
Business span pack	5 journeys manual spans	RCA drill uses correct spans
Tail sampling + hash LB	Policy errors/slow	Error trace capture ≥95% vs metrics
Pipeline isolation	Separate logs vs traces export	Slow Loki does not drop traces
Attr allowlist	Redact PII	0 findings on sample scan
Span budget	Max spans/trace alert	p95 spans/trace within threshold

Deliverables: sampling policy doc; cost before/after.

90 Days — Platform Maturity¶

Experiment	How	Success metric
OTel Operator at scale	Auto-instr tier-1	Time-to-instrument new service <1 day
Chaos collector	Kill gateway pods	No multi-minute full gap
Light schema registry	Semconv review board	Dashboard break rate down
AIOps-ready export	Stable attrs + Kafka	Ch08/10 consumers can read
SLO for telemetry	% successful export	Error budget for telemetry pipeline

Suggested north-star:
  - % critical requests with complete trace tree
  - Collector drop rate (target ~0 under normal)
  - p95 time from deploy → useful spans in Tempo
  - Telemetry cost / 1M spans
  - MTTD for "broken propagation" via synthetic

KEY IDEA

OTel is mature when lost instrumentation is treated as an incident — because you are flying without a black box.

24. Production Review¶

Potential issues:

Tail sampling needs consistent hashing: When scaling the gateway, spans of the same trace must reach the same replica. Without consistent hash routing → wrong sampling decisions. Use a load balancer with hash-based routing on the traceId header.
Persistent queue prevents data loss: If the collector crashes, the memory queue is lost. Use the file_storage extension for a WAL-backed queue.
Exemplars need config in both SDK AND Prometheus: Enabling exemplars at the SDK is not enough. Prometheus also needs --enable-feature=exemplar-storage.

Scores¶

Criterion	Score
Technical Accuracy	9.7/10
Production Readiness	9.6/10
Depth	9.7/10
Practical Value	9.8/10
Cost Awareness	9.7/10

Chapter 02 — OpenTelemetry¶

Prerequisites¶

Related Documents¶

Next Reading¶

1. Why OpenTelemetry?¶

The Problem Before OTel¶

What OTel Solves¶

OTel vs Other Collection Options¶

2. OTel Components Overview¶

3. OTLP Protocol¶

Protocol Variants¶

OTLP Data Model — Trace (simplified)¶

4. The OTel Collector Deep Dive¶

Internal Architecture¶

Collector Distributions¶

5. Receiver Configuration¶

OTLP Receiver¶

Prometheus Receiver (pull-based)¶

Kafka Receiver¶

6. Processor Configuration¶

Memory Limiter Processor (ALWAYS FIRST)¶

Batch Processor¶

Transform Processor¶

Filter Processor¶

Tail Sampling Processor¶

Attributes Processor¶

7. Exporter Configuration¶

OTLP Exporter (→ Tempo for traces)¶

Prometheus Remote Write Exporter (→ Prometheus)¶

Loki Exporter (→ Loki for logs)¶

Kafka Exporter (→ AIOps pipeline)¶

File Storage Extension (persistent queue)¶

8. Pipeline Definition¶

9. Deployment Patterns¶

Pattern 1: Agent + Gateway (Recommended for Production)¶

Pattern 2: Sidecar (For specialized services)¶

10. Kubernetes Operator¶

Installing the Operator¶

OpenTelemetryCollector CRD¶

Auto-Instrumentation CRD¶

11. Fluent Bit vs OTel Collector (and Vector / Alloy)¶

Decision Matrix¶

Pros / cons (edge tools vs each other)¶

Hybrid Pattern (recommended)¶

12. Production Best Practices¶

Resource Limits (Production Sizing)¶

HorizontalPodAutoscaler for Gateway¶

13. Common Mistakes¶

14. Monitoring the Collector¶

Alerting Rules¶

15. Scaling¶

Scaling Bottlenecks¶

Target Allocator (Prometheus scraping at scale)¶

16. Security¶

mTLS with cert-manager¶

Secrets Management¶

17. Cost¶

OTel Collector Resource Cost¶

Data Volume Impact (Cost Savings)¶

18. Production Problem-Solving Mindset¶

18.1 Auto vs Manual Instrumentation — Choose by Questions, Not Fashion¶

18.2 Context Propagation Breaks — "Truncated Trace" Is a Production Bug¶

18.3 Collector SPOF — Agent Dies, You Are Blind; Gateway Dies, You Lose Intentional Sampling¶

18.4 Processor Ordering — Why Order Matters More Than "Having Processors"¶

18.5 Problem-Solving Loop for "No Data Seen"¶

19. Real-World Edge Cases¶

EC-01 — Auto-instrument "has spans" but lacks business context¶

EC-02 — Manual spans too fine-grained → cost + noise¶

EC-03 — traceparent stripped by API gateway¶

EC-04 — Async messaging loses context¶

EC-05 — Single-replica collector gateway¶

EC-06 — Wrong tail sampling when scaling out¶

EC-07 — memory_limiter not first in pipeline¶

EC-08 — Filter after batch / wrong condition → swallows error logs¶

EC-09 — Java agent version pin causes deadlock / permgen issues¶

EC-10 — Fan-out exporter: one slow backend backpressures the whole pipeline¶

EC-11 — Missing resource attributes service.name¶

EC-12 — PII leaks into baggage / attributes¶

20. Decision Trees¶

20.1 Auto, Manual, or Hybrid?¶

EC-03 — `traceparent` stripped by API gateway¶

EC-11 — Missing resource attributes `service.name`¶