// HACKER NEWS — CYBERSECURITY
How we configured OpenTelemetry logs in Rails
Recently, we needed to add observability to a Rails project without getting
locked into a single vendor. OpenTelemetry was the natural choice.
It generates three kinds of telemetry data, called signals: logs, metrics,
and traces. Each signal is independent, so you can adopt one without the others.
All of these signals travel in a standard, vendor-agnostic format known as OTLP
(OpenTelemetry Protocol). Because this format is standard across the industry,
we can switch backends (like Datadog, New Relic, or Grafana Cloud) in the future
without rewriting any application code.
In this post, we will explain how we configured the OpenTelemetry Ruby SDK to
export logs directly to our vendor, Grafana Cloud. We will also discuss a few
issues we encountered along the way and the upstream fixes we contributed.
The standard OpenTelemetry deployment involves running an OpenTelemetry
Collector alongside your application. The Collector is a separate process,
usually a sidecar container or a service on the same host. Your application
sends telemetry to it over the local network, and the Collector then batches
that data and forwards it to the vendor.
To keep our infrastructure simple, we bypassed the Collector and exported logs
directly from the Ruby SDK to Grafana Cloud. The
Scout APM logging gem uses
a similar approach and sends logs directly from the app.
Our log volume is small, so the SDK's built-in batching is enough. A Collector
is still the better choice if you need buffering, sampling, or scrubbing outside
the app.
OpenTelemetry is highly modular, so each gem handles a specific responsibility:
With these gems installed, the exporter needs to know where to send the data.
Set your vendor's endpoint and authentication token as environment variables
(the SDK automatically detects
standard OTLP exporter environment variables):
Finally, we need to initialize the SDK so it starts collecting and exporting
data. Create an initializer (config/initializers/opentelemetry.rb):