- Open Telemetry - Deep Dive into Open Telemetry for .NET
- Open Telemetry - Configuring Open Telemetry for ASP.NET Core
- Open Telemetry - Exporting Open Telemetry Data to Jaeger
- Open Telemetry - Optimally Configuring Open Telemetry Tracing for ASP.NET Core
Configuring tracing in Open Telemetry for ASP.NET Core can be a fairly simple process but never accept the defaults! There is always more we can do to make improvements.
In this post, I'll show you how you can take the simplest setup for Open Telemetry tracing I showed you in 'Configuring Open Telemetry for ASP.NET Core' and move to a more fully featured example.
Simplest Setup
Here is a reminder of the simple setup I showed you in 'Configuring Open Telemetry for ASP.NET Core':
public virtual void ConfigureServices(
IServiceCollection services,
IWebHostEnvironment webHostEnvironment)
{
// ...omitted
services.AddOpenTelemetryTracing(
builder =>
{
builder
.SetResourceBuilder(ResourceBuilder
.CreateDefault()
.AddService(webHostEnvironment.ApplicationName))
.AddAspNetCoreInstrumentation();
if (webHostEnvironment.IsDevelopment())
{
builder.AddConsoleExporter(
options => options.Targets = ConsoleExporterOutputTargets.Debug);
}
});
}
And the tracing output you can expect for a request/response cycle:
Activity.Id: 00-dde96d459fee4144a83818e054e221b1-cac69896c1bcd14f-01
Activity.DisplayName: /favicon-32x32.png
Activity.Kind: Server
Activity.StartTime: 2021-02-01T10:28:25.4637044Z
Activity.Duration: 00:00:00.0086712
Activity.TagObjects:
http.host: localhost:5001
http.method: GET
http.path: /favicon-32x32.png
http.url: https://localhost:5001/favicon-32x32.png
http.user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36
http.status_code: 200
otel.status_code: UNSET
service.name: ApiTemplate
service.instance.id: defe9269-04f2-4b49-a05c-ebddf2112993
telemetry.sdk.name: opentelemetry
telemetry.sdk.language: dotnet
telemetry.sdk.version: 1.0.0.0
The Kitchen Sink Setup
This time we're going to setup a more advanced configuration. We're going to start off by adding a lot more information to our ResourceBuilder we pass to the SetResourceBuilder function above.
private static ResourceBuilder GetResourceBuilder(IWebHostEnvironment webHostEnvironment)
{
var version = Assembly
.GetExecutingAssembly()
.GetCustomAttribute<AssemblyFileVersionAttribute>()!
.Version
ResourceBuilder
.CreateEmpty()
.AddService(webHostEnvironment.ApplicationName, serviceVersion: version)
.AddAttributes(
new KeyValuePair<string, object>[]
{
new("deployment.environment", webHostEnvironment.EnvironmentName),
new("host.name", Environment.MachineName),
})
.AddEnvironmentVariableDetector();
}
This time we want to start with an empty resource builder calling CreateEmpty. We then add the application name and version which we can retrieve from the current assembly. You may have multiple versions of your application running over time and its important to have a way to differentiate between them.
We then add a few attributes to every span including the environment name and machine name. The attribute names here are standardised as defined by the Open Telemetry specification. I decided that knowing the environment the application is running in and the machine name is important to know when troubleshooting issues.
Finally, we add an environment variable detector which we can use to add further attributes to every span using environment variables. Using ResourceBuilder.CreateDefault already included this in the simple example above but since we started with an empty resource builder we need to add it explicitly. Here is a PowerShell example of how you can add add additional attributes to every span using the OTEL_RESOURCE_ATTRIBUTES environment variable:
$env:OTEL_RESOURCE_ATTRIBUTES = 'key1=value1,key2=value2'
We can now plug the GetResourceBuilder into our code below and add a few more goodies:
public virtual void ConfigureServices(
IServiceCollection services,
IWebHostEnvironment webHostEnvironment)
{
// ...omitted
services.AddOpenTelemetryTracing(
builder =>
{
builder
.SetResourceBuilder(GetResourceBuilder(webHostEnvironment))
.AddAspNetCoreInstrumentation(
options =>
{
options.Enrich = Enrich;
options.RecordException = true;
});
if (webHostEnvironment.IsDevelopment())
{
builder.AddConsoleExporter(
options => options.Targets = ConsoleExporterOutputTargets.Debug);
}
});
}
private static void Enrich(Activity activity, string eventName, object obj)
{
if (obj is HttpRequest request)
{
var context = request.HttpContext;
activity.AddTag("http.flavor", GetHttpFlavour(request.Protocol));
activity.AddTag("http.scheme", request.Scheme);
activity.AddTag("http.client_ip", context.Connection.RemoteIpAddress);
activity.AddTag("http.request_content_length", request.ContentLength);
activity.AddTag("http.request_content_type", request.ContentType);
}
else if (obj is HttpResponse response)
{
activity.AddTag("http.response_content_length", response.ContentLength);
activity.AddTag("http.response_content_type", response.ContentType);
}
}
public static string GetHttpFlavour(string protocol)
{
if (HttpProtocol.IsHttp10(protocol))
{
return "1.0";
}
else if (HttpProtocol.IsHttp11(protocol))
{
return "1.1";
}
else if (HttpProtocol.IsHttp2(protocol))
{
return "2.0";
}
else if (HttpProtocol.IsHttp3(protocol))
{
return "3.0";
}
throw new InvalidOperationException($"Protocol {protocol} not recognised.");
}
Next, we configure AddAspNetCoreInstrumentation to enrich the spans with additional information about the current request, response using standardised attributes. Finally, we record details of exceptions from our controllers which would otherwise be lost. This outputs the following:
Activity.Id: 00-3d0f70e71a8e6e5e87f156bdcf94b8c9-ccdd8d23a2e3ba93-01
Activity.ActivitySourceName: OpenTelemetry.Instrumentation.AspNetCore
Activity.DisplayName: /favicon-32x32.png
Activity.Kind: Server
Activity.StartTime: 2022-02-03T10:52:47.6513334Z
Activity.Duration: 00:00:00.0077181
Activity.TagObjects:
http.host: localhost:5001
http.method: GET
http.target: /favicon-32x32.png
http.url: https://localhost:5001/favicon-32x32.png
http.user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36
http.flavor: 2.0
http.scheme: https
http.client_ip: ::1
http.request_content_length:
http.request_content_type:
http.status_code: 200
otel.status_code: UNSET
http.response_content_length: 628
http.response_content_type: image/png
deployment.environment: Development
host.name: REHANS-MACHINE
service.name: ApiTemplate
service.version: 5.1.1.0
service.instance.id: 4e364d08-4965-4d83-8afa-70769074ab0d
This time around you can see we've collected a lot more information. Now, this may not be 'optimal' for your application. Collecting additional information comes at a performance and monetary cost, so its up to you to judge what extra information is useful to you but I think most of the above is pretty essential basic information that would be valuable while debugging any issues.
Wrapping Up
In this post, I showed a simple example showing how you can configure Open Telemetry tracing and then went on to show a more advanced real world example.
Open Telemetry is gaining popularity and traction with even GitHub adopting it. So far in this blog series we've only discussed the basics of Open Telemetry and tracing in particular. When Open Telemetry metrics and logs comes out of alpha/beta, I'll write another post discussing configuring those.
6 Comments
Comment on GitHub