AHdark Blog

In previous posts,¹ I mentioned that in Rust, tracing is often used alongside opentelemetry to build a local-cluster model. In this setup, tracing is responsible for generating and structuring local spans, while opentelemetry takes care of propagating context across service boundaries.

However, I’ve come to realize that this explanation alone doesn’t quite capture what a real-world implementation of tracing injection looks like.

In this post, I’ll walk you through a more elegant approach to integrating tracing with opentelemetry. This method goes beyond simply extracting metadata—by leveraging opentelemetry::propagation, we can achieve a much cleaner and more robust solution.

The problem#

In complex applications, remote calls are common. In a microservices architecture, services communicate with each other through RPC. In message queue models, we often need to propagate context between the publish and consume stages. In these scenarios, it's crucial to pass the tracing context across process boundaries.

In the previous post, I used manual metadata extraction to propagate tracing context. This approach is verbose and error-prone. For example:

req.metadata_mut().insert(
    RPC_TRACE_ID,
    span.context()          // get the opentelemetry context
        .span()             // get the opentelemetry span
        .span_context()     // get the opentelemetry span context, including informations that we need to transmit
        .trace_id()
        .to_string()
        .parse()
        .unwrap(),
);

This method makes us the metadata extractor, manually accessing and parsing data from the span context. Instead of doing this, the opentelemetry::propagation module offers a more elegant solution using the Injector and Extractor traits. Injector is used to insert context into a data structure, and Extractor retrieves it. With these traits, context propagation becomes cleaner and more maintainable.

The solution#

To use opentelemetry::propagation, we need to implement the Injector and Extractor traits for our metadata type.

Implementation based on orphan rule#

According to the orphan rule, sometimes we need to create a new type. Currently I'm working on a tracing middleware for volo-grpc and a message wrapper for broccoli-queue, and that's what problem I encountered.

#[derive(Debug, Clone, PartialEq, serde::Serialize, serde::Deserialize)]
pub struct MessageWithMetadata<T> {
    metadata: HashMap<String, String>,
    payload: T,
}

impl<T> MessageWithMetadata<T> {
    pub fn new(payload: T) -> Self {
        Self {
            metadata: HashMap::new(),
            payload,
        }
    }
}

impl<T> opentelemetry::propagation::Injector for MessageWithMetadata<T> {
    fn set(
        &mut self,
        key: &str,
        value: String,
    ) {
        self.metadata.insert(key.to_string(), value);
    }
}

impl<T> opentelemetry::propagation::Extractor for MessageWithMetadata<T> {
    fn get(
        &self,
        key: &str,
    ) -> Option<&str> {
        self.metadata.get(key).map(|s| s.as_str())
    }

    fn keys(&self) -> Vec<&str> {
        self.metadata.keys().map(|s| s.as_str()).collect()
    }
}

It’s important to note that I implemented the metadata type myself because broccoli-queue doesn’t support complete metadata transmission out of the box. As a result, this wrapper isn’t just for enabling the propagator—it also fulfills the need to carry metadata alongside the message.

Use the Injector and Extractor#

With the Injector and Extractor traits implemented, we can now propagate context in a structured and type-safe way.

The TextMapPropagator trait provides the core methods inject and extract for context propagation. However, you might notice that using these methods directly often results in no context being injected or extracted. This is because they rely on opentelemetry’s internal context, which is separate from the context managed by tracing.

Since tracing stores context in tracing::Dispatch, we must bridge this gap explicitly. To propagate context properly, we need to extract an opentelemetry::span::Span from the current tracing::Span using the tracing_opentelemetry::OpenTelemetrySpanExt trait. We then pass the resulting opentelemetry::Context to the appropriate propagation methods—inject_context for injection and extract_with_context for extraction.

The updated code looks like this:

// Inject context into the outgoing message
let cx = tracing::Span::current().context();

let mut message_wrap = MessageWithMetadata::new(payload);
opentelemetry::global::get_text_map_propagator(|propagator| {
    propagator.inject_context(&cx, &mut message_wrap);
});

// Extract context from the incoming message
let mut cx = tracing::Span::current().context();
opentelemetry::global::get_text_map_propagator(|propagator| {
    cx = propagator.extract(&message.payload);
});
tracing::span::Span::current().set_parent(cx);

Conclusion#

In this post, I showed you how to connect tracing with opentelemetry in a more elegant way using the opentelemetry::propagation module. By implementing the Injector and Extractor traits for our metadata type, we can easily propagate context across process boundaries without manually extracting metadata.

Comparing to the previous method, this approach is cleaner and more maintainable. RemoteSpanContext have many fields, and we don't need to access them manually anymore. We can just use the opentelemetry::propagation module to do that. This is a great improvement for observability in Rust applications.