--- title: "Optimally Configuring Open Telemetry Tracing for ASP.NET Core - Muhammad Rehan Saeed" description: "How to optimally configure Open Telemetry traces for ASP.NET Core enriched with lots of extra information." canonical_url: "https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/" last_updated: "2022-02-03T00:00:00+00:00" meta: author: "Muhammad Rehan Saeed" keywords: "Open Telemetry,.NET,.NET Core,ASP.NET Core,Metrics,Logging,Tracing,Distributed Tracing,Span,Activity" "og:description": "How to optimally configure Open Telemetry traces for ASP.NET Core enriched with lots of extra information." "og:image": "https://rehansaeed.com/_og/s/o_65142fa5f44add92.png" "og:image:alt": "Optimally Configuring Open Telemetry Tracing for ASP.NET Core - Muhammad Rehan Saeed" "og:image:height": "630" "og:image:type": image/png "og:image:width": "1200" "og:title": "Optimally Configuring Open Telemetry Tracing for ASP.NET Core" "twitter:image": "https://rehansaeed.com/_og/s/o_65142fa5f44add92.png" "twitter:image:alt": "Optimally Configuring Open Telemetry Tracing for ASP.NET Core - Muhammad Rehan Saeed" --- > Index of every page on this site: [llms.txt](https://rehansaeed.com/llms.txt) # [Optimally Configuring Open Telemetry Tracing for ASP.NET Core](https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/) Posted 3 February 2022 - **4 min read** ![Open Telemetry](https://rehansaeed.com/_ipx/f_png&s_2560x1440/images/hero/Open-Telemetry-1600x900.png) 1. [Open Telemetry - Deep Dive into Open Telemetry for .NET](https://rehansaeed.com/deep-dive-into-open-telemetry-for-net/) 2. [Open Telemetry - Configuring Open Telemetry for ASP.NET Core](https://rehansaeed.com/open-telemetry-for-asp-net-core/) 3. [Open Telemetry - Exporting Open Telemetry Data to Jaeger](https://rehansaeed.com/exporting-open-telemetry-data-to-jaeger/) 4. [Open Telemetry - Optimally Configuring Open Telemetry Tracing for ASP.NET Core](https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/) Configuring tracing in Open Telemetry for ASP.NET Core can be a fairly simple process but never accept the defaults! There is always more we can do to make improvements. In this post, I'll show you how you can take the simplest setup for Open Telemetry tracing I showed you in ['Configuring Open Telemetry for ASP.NET Core'](https://rehansaeed.com/open-telemetry-for-asp-net-core/) and move to a more fully featured example. # Simplest Setup Link to this section Here is a reminder of the simple setup I showed you in ['Configuring Open Telemetry for ASP.NET Core'](https://rehansaeed.com/open-telemetry-for-asp-net-core/): ```cs public virtual void ConfigureServices( IServiceCollection services, IWebHostEnvironment webHostEnvironment) { // ...omitted services.AddOpenTelemetryTracing( builder => { builder .SetResourceBuilder(ResourceBuilder .CreateDefault() .AddService(webHostEnvironment.ApplicationName)) .AddAspNetCoreInstrumentation(); if (webHostEnvironment.IsDevelopment()) { builder.AddConsoleExporter( options => options.Targets = ConsoleExporterOutputTargets.Debug); } }); } ``` And the tracing output you can expect for a request/response cycle: ```text Activity.Id: 00-dde96d459fee4144a83818e054e221b1-cac69896c1bcd14f-01 Activity.DisplayName: /favicon-32x32.png Activity.Kind: Server Activity.StartTime: 2021-02-01T10:28:25.4637044Z Activity.Duration: 00:00:00.0086712 Activity.TagObjects: http.host: localhost:5001 http.method: GET http.path: /favicon-32x32.png http.url: https://localhost:5001/favicon-32x32.png http.user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36 http.status_code: 200 otel.status_code: UNSET service.name: ApiTemplate service.instance.id: defe9269-04f2-4b49-a05c-ebddf2112993 telemetry.sdk.name: opentelemetry telemetry.sdk.language: dotnet telemetry.sdk.version: 1.0.0.0 ``` # The Kitchen Sink Setup Link to this section This time we're going to setup a more advanced configuration. We're going to start off by adding a lot more information to our `ResourceBuilder` we pass to the `SetResourceBuilder` function above. ```cs private static ResourceBuilder GetResourceBuilder(IWebHostEnvironment webHostEnvironment) { var version = Assembly .GetExecutingAssembly() .GetCustomAttribute()! .Version ResourceBuilder .CreateEmpty() .AddService(webHostEnvironment.ApplicationName, serviceVersion: version) .AddAttributes( new KeyValuePair[] { new("deployment.environment", webHostEnvironment.EnvironmentName), new("host.name", Environment.MachineName), }) .AddEnvironmentVariableDetector(); } ``` This time we want to start with an empty resource builder calling `CreateEmpty`. We then add the application name and version which we can retrieve from the current assembly. You may have multiple versions of your application running over time and its important to have a way to differentiate between them. We then add a few attributes to every span including the environment name and machine name. The attribute names here are [standardised](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/README.md) as defined by the Open Telemetry specification. I decided that knowing the environment the application is running in and the machine name is important to know when troubleshooting issues. Finally, we add an environment variable detector which we can use to add further attributes to every span using environment variables. Using `ResourceBuilder.CreateDefault` already included this in the simple example above but since we started with an empty resource builder we need to add it explicitly. Here is a PowerShell example of how you can add add additional attributes to every span using the `OTEL_RESOURCE_ATTRIBUTES` environment variable: ```powershell $env:OTEL_RESOURCE_ATTRIBUTES = 'key1=value1,key2=value2' ``` We can now plug the `GetResourceBuilder` into our code below and add a few more goodies: ```cs public virtual void ConfigureServices( IServiceCollection services, IWebHostEnvironment webHostEnvironment) { // ...omitted services.AddOpenTelemetryTracing( builder => { builder .SetResourceBuilder(GetResourceBuilder(webHostEnvironment)) .AddAspNetCoreInstrumentation( options => { options.Enrich = Enrich; options.RecordException = true; }); if (webHostEnvironment.IsDevelopment()) { builder.AddConsoleExporter( options => options.Targets = ConsoleExporterOutputTargets.Debug); } }); } private static void Enrich(Activity activity, string eventName, object obj) { if (obj is HttpRequest request) { var context = request.HttpContext; activity.AddTag("http.flavor", GetHttpFlavour(request.Protocol)); activity.AddTag("http.scheme", request.Scheme); activity.AddTag("http.client_ip", context.Connection.RemoteIpAddress); activity.AddTag("http.request_content_length", request.ContentLength); activity.AddTag("http.request_content_type", request.ContentType); } else if (obj is HttpResponse response) { activity.AddTag("http.response_content_length", response.ContentLength); activity.AddTag("http.response_content_type", response.ContentType); } } public static string GetHttpFlavour(string protocol) { if (HttpProtocol.IsHttp10(protocol)) { return "1.0"; } else if (HttpProtocol.IsHttp11(protocol)) { return "1.1"; } else if (HttpProtocol.IsHttp2(protocol)) { return "2.0"; } else if (HttpProtocol.IsHttp3(protocol)) { return "3.0"; } throw new InvalidOperationException($"Protocol {protocol} not recognised."); } ``` Next, we configure `AddAspNetCoreInstrumentation` to enrich the spans with additional information about the current request, response using standardised attributes. Finally, we record details of exceptions from our controllers which would otherwise be lost. This outputs the following: ```text Activity.Id: 00-3d0f70e71a8e6e5e87f156bdcf94b8c9-ccdd8d23a2e3ba93-01 Activity.ActivitySourceName: OpenTelemetry.Instrumentation.AspNetCore Activity.DisplayName: /favicon-32x32.png Activity.Kind: Server Activity.StartTime: 2022-02-03T10:52:47.6513334Z Activity.Duration: 00:00:00.0077181 Activity.TagObjects: http.host: localhost:5001 http.method: GET http.target: /favicon-32x32.png http.url: https://localhost:5001/favicon-32x32.png http.user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36 http.flavor: 2.0 http.scheme: https http.client_ip: ::1 http.request_content_length: http.request_content_type: http.status_code: 200 otel.status_code: UNSET http.response_content_length: 628 http.response_content_type: image/png deployment.environment: Development host.name: REHANS-MACHINE service.name: ApiTemplate service.version: 5.1.1.0 service.instance.id: 4e364d08-4965-4d83-8afa-70769074ab0d ``` This time around you can see we've collected a lot more information. Now, this may not be 'optimal' for your application. Collecting additional information comes at a performance and monetary cost, so its up to you to judge what extra information is useful to you but I think most of the above is pretty essential basic information that would be valuable while debugging any issues. # Wrapping Up Link to this section In this post, I showed a simple example showing how you can configure Open Telemetry tracing and then went on to show a more advanced real world example. Open Telemetry is gaining popularity and traction with even [GitHub adopting it](https://github.blog/2021-05-26-why-and-how-github-is-adopting-opentelemetry/). So far in this blog series we've only discussed the basics of Open Telemetry and tracing in particular. When Open Telemetry metrics and logs comes out of alpha/beta, I'll write another post discussing configuring those. ## Web Mentions [What's this?](https://en.wikipedia.org/wiki/Webmention) ### 0 Linked ### 0 Replies ## GitHub Comments ### 6 Comments [Comment on GitHub](https://github.com/RehanSaeed/rehansaeed.github.io/issues/607) [![ZedZipDev](https://avatars.githubusercontent.com/u/25686753?v=4)](https://github.com/ZedZipDev) ### [ZedZipDev](https://github.com/ZedZipDev) 4 February 2022 Hi, thank you for the great article! One question. I have multiple c# applications: NET Core, legacy WinForms apps. Is it possible to use OpenTelemetry in WinForms apps? How to? [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 4 February 2022 > Hi, thank you for the great article! One question. I have multiple c# applications: NET Core, legacy WinForms apps. Is it possible to use OpenTelemetry in WinForms apps? How to? You certainly can. With ASP.NET Core, there are packages which setup some basic telemetry for you. With desktop apps, you'd have to do all that yourself. Take a look at my first blog post in the series, that should be enough to get you going. [![bmcclory](https://avatars.githubusercontent.com/u/4788982?v=4)](https://github.com/bmcclory) ### [bmcclory](https://github.com/bmcclory) 17 May 2022 Your example Enrich function demonstrates reading `HttpContext.User` during the "OnStartActivity" callback, to attach "enduser" attributes (or tags) to the `Activity`. A couple questions about this: - "OnStartActivity" executes *very* early in the ASP.NET Core request pipeline, long before application authentication middleware has executed. I believe the `User` principal will always be anonymous at this point. - It'd probably work correctly if you read `User` during the "OnStopActivity" callback instead, but... - The [Open Telemetry specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md#general-identity-attributes) recommends propagating "enduser" attributes via the Baggage mechanism: > These attributes describe the authenticated user driving the user agent making requests to the instrumented system. It is expected this information would be propagated unchanged from node-to-node within the system using the Baggage mechanism. These attributes should not be used to record system-to-system authentication attributes. A real-world scenario: An end user authenticates with a front-end ASP.NET Core application, which invokes multiple backend systems using application credentials. The "enduser" details for those backend spans should be the original end user (propagated via Baggage) and not the front-end application's client ID. Now we have a conundrum. "OnStopActivity" is too late in the request execution to meaningfully attach Baggage to an activity -- the activity is already (nearly) complete and that Baggage won't be propagated. But "OnStartActivity" is too early to resolve the enduser details! I *think* the solution here is to manage "enduser' Baggage somewhere else -- outside of this Activity Enrich callback. Probably a middleware component that runs immediately after Authentication. In a system where all end-user traffic flows through a gateway, that's likely the only place that needs such a middleware. Backend ASP.NET Core apps (behind the gateway) might enrich all their Activities with that Baggage data -- e.g. using a generalized EnrichingProcessor [like this OTEL .NET sample](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/extending-the-sdk/MyEnrichingProcessor.cs) -- rather than only enriching the activities produced by the AspNetCoreInstrumentation. But I'm pretty new to all of this and am still just getting started with OTEL... [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 20 May 2022 [@bmcclory](https://github.com/bmcclory) I wrote that extra bit of extra code for illustration but I think you're right, we'd need to use a filter to add tags or baggage. I'll update my post to remove that bit of code. [![bmcclory](https://avatars.githubusercontent.com/u/4788982?v=4)](https://github.com/bmcclory) ### [bmcclory](https://github.com/bmcclory) 20 May 2022 Sounds good. Great blog series by the way -- was one of the first places I landed when I began my journey into OTEL .NET and was an excellent orientation. Thanks! [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 21 May 2022 Thanks [@bmcclory](https://github.com/bmcclory). This is another excuse for a new post. ```json {"@context":"https://schema.org","@graph":[{"@id":"https://rehansaeed.com/#website","@type":"WebSite","description":"Software Developer at Microsoft, YouTuber, Open Source Contributor and Blogger","inLanguage":"en-GB","name":"Muhammad Rehan Saeed","url":"https://rehansaeed.com/","author":{"@id":"https://rehansaeed.com/#identity"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://rehansaeed.com/?search={search_term_string}"},"query-input":"required name=search_term_string"}],"publisher":{"@id":"https://rehansaeed.com/#publisher"}},{"@id":"https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/#webpage","@type":"WebPage","dateModified":"2022-02-03T00:00:00+00:00","datePublished":"2022-02-03T00:00:00+00:00","description":"How to optimally configure Open Telemetry traces for ASP.NET Core enriched with lots of extra information.","name":"Optimally Configuring Open Telemetry Tracing for ASP.NET Core","url":"https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/","breadcrumb":{"@id":"https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/#breadcrumb"},"isPartOf":{"@id":"https://rehansaeed.com/#website"},"potentialAction":[{"@type":"ReadAction","target":["https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/"]}]},{"@id":"https://rehansaeed.com/#identity","@type":"Person","name":"Muhammad Rehan Saeed","url":"https://rehansaeed.com/about/","image":{"@id":"https://rehansaeed.com/#/schema/image/1"},"sameAs":["https://github.com/RehanSaeed","https://www.linkedin.com/in/muhammad-rehan-saeed","https://stackoverflow.com/users/1212017/muhammad-rehan-saeed","https://www.youtube.com/c/MuhammadRehanSaeed","https://www.twitch.tv/rehansaeeduk"]},{"@id":"https://rehansaeed.com/#publisher","@type":"Organization","logo":"https://rehansaeed.com/images/schema/Publisher-600x60.png","name":"Muhammad Rehan Saeed","url":"https://rehansaeed.com"},{"@id":"https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/#article","@type":"Article","dateModified":"2022-02-03T00:00:00+00:00","datePublished":"2022-02-03T00:00:00+00:00","description":"How to optimally configure Open Telemetry traces for ASP.NET Core enriched with lots of extra information.","headline":"Optimally Configuring Open Telemetry Tracing for ASP.NET Core","inLanguage":"en-GB","url":"https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/","author":{"@id":"https://rehansaeed.com/#identity"},"image":[{"@type":"ImageObject","url":"https://rehansaeed.com/images/hero/Open-Telemetry-1600x900.png","caption":"Open Telemetry","width":"1600","height":"900"}],"isPartOf":{"@id":"https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/#webpage"},"keywords":["Open Telemetry",".NET",".NET Core","ASP.NET Core","Metrics","Logging","Tracing","Distributed Tracing","Span","Activity"],"mainEntityOfPage":{"@id":"https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/#webpage"},"publisher":{"@id":"https://rehansaeed.com/#publisher"}},{"@id":"https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/#breadcrumb","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","name":"Blog","item":"https://rehansaeed.com/","position":1},{"@type":"ListItem","name":"Optimally Configuring Open Telemetry Tracing for ASP.NET Core","item":"https://rehansaeed.com/optimally-configuring-open-telemetry-tracing-for-asp-net-core/","position":2}]},{"@id":"https://rehansaeed.com/#/schema/image/1","@type":"ImageObject","contentUrl":"https://rehansaeed.com/images/author/Muhammad-Rehan-Saeed/Logo-512x512.png","inLanguage":"en-GB","url":"https://rehansaeed.com/images/author/Muhammad-Rehan-Saeed/Logo-512x512.png"}]} ``` ## Sitemap See the full [sitemap](https://rehansaeed.com/sitemap.md) for all pages.