Exporting Open Telemetry Data to Jaeger

Open Telemetry
  1. Open Telemetry - Deep Dive into Open Telemetry for .NET
  2. Open Telemetry - Configuring Open Telemetry for ASP.NET Core
  3. Open Telemetry - Exporting Open Telemetry Data to Jaeger
  4. Open Telemetry - Optimally Configuring Open Telemetry Tracing for ASP.NET Core

As I talked about in my first post, the end goal is to get nice visualisations from our Open Telemetry data, so we can spot patterns and learn something from the behaviours of our applications.

In this post, I'll show how we can export the Open Telemetry traces, logs and metrics that we've collected to Jaeger and view them in the Jaeger dashboard.

Open Telemetry Protocol (OTLP)

There are actually two methods of exporting our telemetry to Jaeger. The first uses Jaeger's proprietary protocol and is not something I plan to cover in this post. The second uses the Open Telemetry Protocol (OTLP), which is an open standard that we can use to export Open Telemetry data to any application that supports it.

Jaeger is a pretty complex application that splits it's responsibilities into several separate binaries. It splits the collection of telemetry into a binary called a 'collector'. If we want Jaeger to collect telemetry using the Open Telemetry Protocol, we need to use the Jaeger Open Telemetry collector. I'm not going to cover how we can setup Jaeger in a full production setup. Instead I'll be using the jaegertracing/opentelemetry-all-in-one Docker image which makes running Jaeger with the Open Telemetry collector as easy as running this command:

docker run --name jaeger -p 13133:13133 -p 16686:16686 -p 4317:55680 -d --restart=unless-stopped jaegertracing/opentelemetry-all-in-one

I've also opened up a few ports to the Docker container:

  • 4317 - Open Telemetry Protocol (OTLP) receiver where we expect to receive Open Telemetry data.
  • 16686 - Dashboard where users can see visualisations.
  • 13133 - Jaeger health check.

Warning

The default port used by the Open Telemetry Protocol has recently been changed from 55680 to 4317. This change has not yet been made in Jaeger which uses the older port number.

Exporting from .NET

In the last post I showed how we can collect Open Telemetry data in an ASP.NET Core application. I'm going to build on that example to export to Jaeger. To start with, we'll need to add an additional NuGet package called OpenTelemetry.Exporter.OpenTelemetryProtocol:

<PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.0.1" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.0.1" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc2" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc2" />

Then we can use the AddOtlpExporter method to configure where to export to. In this case, I'm exporting to http://localhost:4317 where port 4317 is the default port used by the Open Telemetry Protocol. Ideally, we'd retrieve this value from configuration but I'm keeping things simple in this example.

services.AddOpenTelemetryTracing(
    builder =>
    {
        builder
            .SetResourceBuilder(ResourceBuilder
                .CreateDefault()
                .AddService(webHostEnvironment.ApplicationName))
            .AddAspNetCoreInstrumentation()
            .AddOtlpExporter(options => options.Endpoint = new Uri("http://localhost:4317"));
        if (webHostEnvironment.IsDevelopment())
        {
            builder.AddConsoleExporter(options => options.Targets = ConsoleExporterOutputTargets.Debug);
        }
    });

The Dashboard

We can now fire up the Jaeger dashboard in our browser which we can access at http://localhost:16686. If we execute some request/response cycles in our application where we have added Open Telemetry support, we can see the telemetry for each request/response:

The Jaeger dashboard showing request/response cycles

If we drill down into a particular request/response trace, we can view the spans (in this simple example, there is only one) and all attributes associated with the span. This is the same data we saw in the debug output from my previous post:

The Jaeger dashboard showing request/response cycles

Up Next

In this post, I've shown how you can quickly fire up Jaeger and how you can get an ASP.NET Core app to export Open Telemetry data to it. In my next final post I'll discuss optimally configuring Open Telemetry for an ASP.NET application.

Web Mentions

What's this?

0 Replies

GitHub Comments

martinmani
(edited )

I ran the docker for jaeger with following command as given in the post
docker run --name jaeger -p 13133:13133 -p 16686:16686 -p 4317:55680 -d --restart=unless-stopped jaegertracing/opentelemetry-all-in-one

and configure the otlp exporter to http://localhost:4317 in my api.
I could see the telemetry in the console of my api.
But I could not see the same of my api in http://localhost:16686/. the only service I could see is 'jaeger-query' in http://localhost:16686.

what could be missing?

martinmani

I changed to the following and found working :)
docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 14250:14250 -p 9411:9411 jaegertracing/all-in-one:1.22

And in the api use AddJaegerExporter() instead of AddOtlpExporter(options => options.Endpoint = new Uri("http://localhost:4317"))

ElectricVampire

I am using Serilog for logging - I enrich all the logs with TraceID and SpanID of Http Request. I am using Prometheus.NET to collect metrics.
How do I leverage OpenTelemetry.NET in this setup?

Can I use OpenTelemetry to collect metrics as well?
Can existing logs from application through Serilog can also be used Telemetry?
I also have one more problem-
I receive messages from Third Party MF system to my kafka queue. Each message has its own unique id. There is .Net core kafka message processor which reads message from Queue and call a multiple Web Apis.
How do I assign unique Id of message to System.Diagnostics.Activity.TraceID in Kafak .Net processor, so any HttpClient call made from this exe to any API get that copied over via request headers.

Alternatively please also let me know if I am approaching the problem in wrong way and there is already a standard solution for this kind of problem.

I really appreciate your time and help in above queries

neoaisac

The series of articles is very interesting and to the point. Great work! However it is mostly focused on the area of OpenTelemetry dedicated to distributed tracing. In the examples above there is no explanation as to how application logs can be also sent and correlated with the tracing capabilities of Jaeger. In fact, when using this configuration along with a ILogger and trying to log information, the Console logger configuration does not log the messages.

How would you achieve correlation between Logs and Traces based in this configuration and using Jaeger?

RehanSaeed

@ElectricVampire Open Telemetry metrics are still a work in progress. You can push logs via Open Telemetry so they show up in tools like Jaeger but this is not currently working out of the box. open-telemetry/opentelemetry-dotnet#1739 talks about adding logging support and how Serilog fits into the picture (hint: we might not need Serilog anymore!). If you have ID's that you want to relate to a trace, you can add them easily as attributes.

@neoaisac I'm glad you found the resource useful! open-telemetry/opentelemetry-dotnet#1739 talks about adding logging support and how Serilog fits into the picture (hint: we might not need Serilog anymore!).

ElectricVampire
(edited )

@RehanSaeed : Thanks for reply,

If you have ID's that you want to relate to a trace, you can add them easily as attributes.

But in that case it won't be used by .net FW. For example if I receive a message from queue -> I read unique ID from message -> I create an activity and assign attribute -> then make a Http web API call within the scope of Activity
According to the code which I understood from HttpClient it will read and copy the Activity.TraceId in header and not from custom attribute.
So is there any way by which .NetFw(specially HttpClient) can be configured to use my unique message Id rather than system generated TraceID)
Thanks again for your time on query.