Skip to main content
Gather relevant insights about your application and modules with custom metrics and telemetry.

Overview

The telemetry package provides observability tooling for Cosmos SDK applications using OpenTelemetry. It offers a unified initialization point for traces, metrics, and logs via the OpenTelemetry declarative configuration API. This package:
  • Initializes OpenTelemetry SDK using YAML configuration files
  • Provides backward compatibility with Cosmos SDK’s legacy go-metrics wrapper API
  • Includes built-in instrumentation for host, runtime, and disk I/O metrics

Quick Start

1. Start a Local Telemetry Backend

2. Create Configuration File

Create an otel.yaml file:

3. Initialize Telemetry

Option A: Environment Variable (Recommended) Set OTEL_EXPERIMENTAL_CONFIG_FILE to your config path. This initializes the SDK before any meters/tracers are created, avoiding atomic load overhead.
Option B: Node Config Directory An empty otel.yaml will now be generated in ~/.<node_home>/config/. Place the desired configuration in otel.yaml. Option C: Programmatic Initialization The SDK will first attempt to initialize via env var, then using the config in the node’s home directory. You may optionally initialize telemetry yourself using the telemetry.InitializeOpenTelemetry function:

Configuration

OpenTelemetry Configuration

The package uses the OpenTelemetry declarative configuration spec. Key sections: For examples containing available options, see the OpenTelemetry configuration examples.

Extensions

The extensions section of the otel.yaml configuration file provides additional features not yet supported by the standard otelconf:

Custom Instruments

Host Instrumentation (host)

Reports host-level metrics using go.opentelemetry.io/contrib/instrumentation/host:
  • CPU usage
  • Memory usage
  • Network I/O

Runtime Instrumentation (runtime)

Reports Go runtime metrics using go.opentelemetry.io/contrib/instrumentation/runtime:
  • Goroutine count
  • GC statistics
  • Memory allocations

Disk I/O Instrumentation (diskio)

Reports disk I/O metrics using gopsutil:
By default, virtual devices (loopback, RAID, partitions) are filtered out on Linux to avoid double-counting I/O.

Propagators

Configure trace context propagation for distributed tracing:

Developer Usage

Using Meters and Tracers

After initialization, use standard OpenTelemetry APIs:

Shutdown

Always call Shutdown() when the application exits:

Legacy API (Deprecated)

The package provides backward-compatible wrappers for github.com/hashicorp/go-metrics. These are deprecated and users should migrate to OpenTelemetry APIs directly.

OpenTelemetry Bridge

Cosmos SDK v0.54.0+ provides a bridge to send existing go-metrics to the meter provider defined in your OpenTelemetry config. To bridge your metrics, set the metrics-sink in app.toml to “otel”.

Legacy Configuration

Legacy Metrics Functions

All are deprecated; prefer OpenTelemetry:

Metrics Sink Types

Best Practices

  1. Use environment variable initialization for production to avoid atomic load overhead
  2. Always call Shutdown() to ensure metrics/traces are flushed
  3. Thread context.Context properly for correct span correlation

Viewing Telemetry Data

With Grafana LGTM running:
  1. Open http://localhost:3000
  2. Use the Drilldown views to explore:
    • Traces: Distributed trace visualization
    • Metrics: Query and dashboard metrics
    • Logs: Structured log search

Cosmos SDK Metrics

The following metrics are emitted from the Cosmos SDK.