--- title: "Dynamically Generating Robots.txt Using ASP.NET MVC - Muhammad Rehan Saeed" description: "How to dynamically generate a robots.txt file using a simple ASP.NET MVC action method and only a few lines of code." canonical_url: "https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/" last_updated: "2015-07-31T00:00:00+00:00" meta: author: "Muhammad Rehan Saeed" keywords: ".NET,ASP.NET MVC,ASP.NET Core Boilerplate,C#,Robots,Search Engine Optimization (SEO),Search Engines,Web Crawlers" "og:description": "How to dynamically generate a robots.txt file using a simple ASP.NET MVC action method and only a few lines of code." "og:image": "https://rehansaeed.com/_og/s/o_32e05965036182b0.png" "og:image:alt": "Dynamically Generating Robots.txt Using ASP.NET MVC - Muhammad Rehan Saeed" "og:image:height": "630" "og:image:type": image/png "og:image:width": "1200" "og:title": "Dynamically Generating Robots.txt Using ASP.NET MVC" "twitter:image": "https://rehansaeed.com/_og/s/o_32e05965036182b0.png" "twitter:image:alt": "Dynamically Generating Robots.txt Using ASP.NET MVC - Muhammad Rehan Saeed" --- > Index of every page on this site: [llms.txt](https://rehansaeed.com/llms.txt) # [Dynamically Generating Robots.txt Using ASP.NET MVC](https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/) Posted 31 July 2015 - **4 min read** ![Robots.txt](https://rehansaeed.com/_ipx/f_jpg&s_2560x1440/images/hero/Robots.txt-1366x768.jpg) - [ASP.NET Core Boilerplate](https://rehansaeed.com/asp-net-mvc-boilerplate/) - Security - [Securing the ASP.NET MVC Web.config (Updated)](https://rehansaeed.com/securing-the-aspnet-mvc-web-config/) - [NWebSec ASP.NET MVC Security Through HTTP Headers](https://rehansaeed.com/nwebsec-asp-net-mvc-security-through-http-headers/) - [Content Security Policy (CSP) for ASP.NET MVC](https://rehansaeed.com/content-security-policy-for-asp-net-mvc/) - Search Engine Optimization (SEO) - [Canonical URL's for ASP.NET MVC](https://rehansaeed.com/canonical-urls-for-asp-net-mvc/) - [Dynamically Generating Robots.txt Using ASP.NET MVC](https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/) - [Internet Favicon Madness (Updated)](https://rehansaeed.com/internet-favicon-madness/) - [Building RSS/Atom Feeds for ASP.NET MVC](https://rehansaeed.com/building-rssatom-feeds-for-asp-net-mvc/) A `robots.txt` file is a simple text file you can place at the root of your site at `http://example.com/robots.txt` to tell search engine robots (also known as web crawlers) how to index your site. The robots know to look for this file at the root of every site before they start indexing the site. If you do not have this file in your site, you will be getting a lot of 404 Not Found errors in your logs. The `robots.txt` uses the [Robots Exclusion Standard](http://en.wikipedia.org/wiki/Robots_exclusion_standard) which is a very simple format that can give robots instructions on what to index and what to skip. A very basic `robots.txt` file looks like this: ```text # Allow all robots to index this site. user-agent: * # Tell all robots not to index any of the pages under the /error path. disallow: /error/ # Tell all robots to index the under the error/foo path. allow: /error/foo/ # Add a link to the site-map. Unfortunately this must be an absolute URL. sitemap: http://example.com/sitemap.xml ``` In the above code, all comments start with the hash character. It tells all robots that they can index everything on the site except pages under the `/error` path because we don't want our error pages showing up in peoples search results. The only exception to that rule is to allow the resources under the `/error/foo` path to be indexed. The last line is interesting and tells robots where to find an XML file called a site-map. A site-map contains a list of URL's to all the pages in the site and is used to give search engines a list of URL's they can go through to index the entire site. It's a great SEO (Search Engine Optimization) technique to give your site a boost in it's search rankings. I will discuss creating a dynamic `sitemap.xml` file for ASP.NET Core in a future post. For now, all you need to know is that the site-map URL has to be an absolute URL according to the specification. This is a pretty terrible decision by whoever created the robots exclusion standard. It's really annoying that when you're creating a site, you have to remember to manually update this URL. If the URL was relative we would not have this problem. # Dynamically Generating a robots.txt File Link to this section Fortunately, it's really easy to dynamically create a `robots.txt` file, which auto-generates the site-map URL using the MVC `UrlHelper`. Take a look at the code below: ```cs public class HomeController : Controller { [Route("robots.txt", Name = "GetRobotsText"), OutputCache(Duration = 86400)] public ContentResult RobotsText() { StringBuilder stringBuilder = new StringBuilder(); stringBuilder.AppendLine("user-agent: *"); stringBuilder.AppendLine("disallow: /error/"); stringBuilder.AppendLine("allow: /error/foo"); stringBuilder.Append("sitemap: "); stringBuilder.AppendLine(this.Url.RouteUrl("GetSitemapXml", null, this.Request.Url.Scheme).TrimEnd('/')); return this.Content(stringBuilder.ToString(), "text/plain", Encoding.UTF8); } [Route("sitemap.xml", Name = "GetSitemapXml"), OutputCache(Duration = 86400)] public ContentResult SitemapXml() { // I'll talk about this in a later blog post. } } ``` I set up a route to the `robots.txt` path at the root of the site in my main `HomeController` and cached the response for a day for better performance (You can and should probably specify a much longer period of time if you know yours won't change). I then go on to append my commands to the StringBuilder. The great thing is that I can easily use the `UrlHelper` to generate a complete absolute URL to the `sitemap.xml` path which is also dynamically generated in much the same way. Finally, I just return the string as plain text using the UTF-8 encoding. Creating a route ending with a file extension is not allowed by default in ASP.NET Core. To get around this security restriction, you need to add the following to the `Web.config` file: ```xml ``` # Conclusion Link to this section Dynamically generating your `robots.txt` file is pretty easy and only takes as many lines of code as you need to write your `robots.txt` file anyway. It also means that you don't need to pollute your project structure with yet another file at the root of it (This problem is fixed in MVC Core, where all static files must be added to the wwwroot folder). You can also dynamically generate your site-map URL so you don't need to remember to update it every time you change the domain. You could argue that performance is an issue when compared to a static `robots.txt` text file but its a matter of a few bytes and if you cache the response with a sufficient time limit then I think that even that problem goes away. Once again, you can find a working example of this and much more using the [ASP.NET Core Boilerplate](https://visualstudiogallery.msdn.microsoft.com/6cf50a48-fc1e-4eaf-9e82-0b2a6705ca7d) project template. ## Web Mentions [What's this?](https://en.wikipedia.org/wiki/Webmention) ### 0 Linked ### 0 Replies ## GitHub Comments ### 12 Comments [Comment on GitHub](https://github.com/RehanSaeed/rehansaeed.github.io/issues/74) [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![vlince](https://camo.githubusercontent.com/aa5a45de3b168d441cf410640666457cd56f6df1fe7f57f5ff20d03884851981/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f38383666323465386334636137336332323031653033633031633236613466633f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/aa5a45de3b168d441cf410640666457cd56f6df1fe7f57f5ff20d03884851981/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f38383666323465386334636137336332323031653033633031633236613466633f733d393626643d6d6d26723d67) **vlince** commented on **2015-08-22 07:48:34** Out of curiosity, what is the added benefit of dynamically creating a `robots.txt` file versus creating a static one in terms of maintenance? I guess my real concern, is having to choose between polluting my project structure with another file versus having to redeploy the application each time I need to add, edit or delete stuff inside the `robots.txt`. What's the lesser of two evils? [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![Muhammad Rehan Saeed](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67) **[Muhammad Rehan Saeed](https://rehansaeed.com/)** commented on **2015-08-22 08:02:08** > Out of curiosity, what is the added benefit of dynamically creating a robots.txt file versus creating a static one in terms of maintenance? > > I guess my real concern, is having to choose between polluting my project structure with another file versus having to redeploy the application each time I need to add, edit or delete stuff inside the robots.txt. > What's the lesser of two evils? Polluting your project structure is a real problem with MVC 5 e.g. if you want to support favicons on all the major platforms then you need to add lots of images to the root or add some meta tags. Thankfully, this problem is solved in MVC 5 with the separate `wwwroot` folder. Another problem is that the site-map URL has to be absolute. You have to manually go in and edit this URL. There are just so many little things like this you have to do to get a site up quickly, that I thought it would be best if I made the computer do the work instead. You have to remember your average developer using the ASP.NET Core Boilerplate project template is not always a web expert. I don't see `robots.txt` as a file you need to edit very often but you are right, you can no longer just edit the file and it means a deployment. Some developers may be less comfortable with this than others. [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 (edited 12 May 2020) [![Fabricio](https://camo.githubusercontent.com/3e9712ce459f9d53ad60cd529f9b4102568a10bcc92e1ab78ba5b4876ebf23e6/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f61613637386363366334356265636232323064623932626462313337306430363f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/3e9712ce459f9d53ad60cd529f9b4102568a10bcc92e1ab78ba5b4876ebf23e6/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f61613637386363366334356265636232323064623932626462313337306430363f733d393626643d6d6d26723d67) **Fabricio** commented on **2015-09-10 04:18:39** So, how about `SitemapXml`? it says "I'll talk about this in a later blog post", so...? [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 (edited 12 May 2020) [![Muhammad Rehan Saeed](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67) **[Muhammad Rehan Saeed](https://rehansaeed.com/)** commented on **2015-09-10 20:55:24** > So, how about `SitemapXml`? it says "I'll talk about this in a later blog post", so...? I've been massively busy lately with a career move. I'll get on it as soon as I can :) [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![Asad Ali](https://camo.githubusercontent.com/ce25d6b15a31c55e13154efccfae8cc00b6c476e38e68e06a09227318703e5b3/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f34383330353464346266636365356266633437336136663366373733636634313f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/ce25d6b15a31c55e13154efccfae8cc00b6c476e38e68e06a09227318703e5b3/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f34383330353464346266636365356266633437336136663366373733636634313f733d393626643d6d6d26723d67) **[Asad Ali](http://www.hoxat.com/)** commented on **2016-03-28 04:08:55** I have used your sitemap generating algorithms and robotsTXT they work both fine at `localhost:3333/sitemap.xml` and `/robots.txt` but after publishing and uploading at hosting service - i have to access those at [www.hoxat.com/sitemap.xml/](http://www.hoxat.com/sitemap.xml/) if i don't add the slash then i get HTML 404 i am using shared hosting Waiting for your kind reply [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![Muhammad Rehan Saeed](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67) **[Muhammad Rehan Saeed](https://rehansaeed.com/)** commented on **2016-03-28 13:57:56** > I have used your sitemap generating algorithms and robotsTXT > they work both fine at `localhost:3333/sitemap.xml` and `/robots.txt` > > but after publishing and uploading at hosting service - i have to access those at [www.hoxat.com/sitemap.xml/](http://www.hoxat.com/sitemap.xml/) > if i don't add the slash then i get HTML 404 > > i am using shared hosting > Waiting for your kind reply Please raise an issue on the GitHub site listing the version of MVC you are using. The `RedirectToCanonicalUrlAttribute` is doing the redirecting. As long as you have the NoTrailingSlashAttribute on these actions it should be ok. [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![Arlan Galvez](https://camo.githubusercontent.com/be89699f5df63d72dbf3aab670b178b41264819b8f8384a02b64a70ec7a64bf4/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f63646235373435323066356263653832366436623765396435666134393162623f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/be89699f5df63d72dbf3aab670b178b41264819b8f8384a02b64a70ec7a64bf4/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f63646235373435323066356263653832366436623765396435666134393162623f733d393626643d6d6d26723d67) **Arlan Galvez** commented on **2016-06-30 20:39:29** Hi, I made all you said to make it work , but I had to put a route config rule because always the response was 404. ``` routes.MapRoute( "Robots.txt", "robots.txt", new { controller = "Home", action = "RobotsText" }, new[] { "VisaCuba.Controllers" }); ``` It is mandatory to get the correct response? I'm using MVC 5. Greetings. [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![Muhammad Rehan Saeed](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67) **[Muhammad Rehan Saeed](https://rehansaeed.com/)** commented on **2016-07-02 11:09:46** > Hi, I made all you said to make it work , but I had to put a route config rule because always the response was 404. > > ``` > routes.MapRoute( > "Robots.txt", > "robots.txt", > new { controller = "Home", action = "RobotsText" }, > new[] { "VisaCuba.Controllers" }); > ``` > > It is mandatory to get the correct response? I'm using MVC 5. > Greetings. Looks like you are not using attribute routing, which is why you needed to add that code. [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![Fabricio Gabrielli da Silva](https://camo.githubusercontent.com/3e9712ce459f9d53ad60cd529f9b4102568a10bcc92e1ab78ba5b4876ebf23e6/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f61613637386363366334356265636232323064623932626462313337306430363f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/3e9712ce459f9d53ad60cd529f9b4102568a10bcc92e1ab78ba5b4876ebf23e6/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f61613637386363366334356265636232323064623932626462313337306430363f733d393626643d6d6d26723d67) **Fabricio Gabrielli da Silva** commented on **2016-07-04 12:35:28** > I've been massively busy lately with a career move. I'll get on it as soon as I can :) Any update on this? [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![Muhammad Rehan Saeed](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/8a07049df41b8a3839a3f4342f09ea8e6e7fa0065667687f04beca2d088a1dca/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f32623266393030653361633066353535316462646537323864336262343935343f733d393626643d6d6d26723d67) **[Muhammad Rehan Saeed](https://rehansaeed.com/)** commented on **2016-07-04 17:28:09** > Any update on this? This was posted some time ago. https://rehansaeed.com/dynamically-generating-sitemap-xml-for-asp-net-mvc/ [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![Ian Buchan](https://camo.githubusercontent.com/cf8d1b07de88b7c791d0cecee177faef31f4b0a78832da6993d7eeb4338934d8/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f65313063313734346530636264366166663466356231643065303337636534653f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/cf8d1b07de88b7c791d0cecee177faef31f4b0a78832da6993d7eeb4338934d8/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f65313063313734346530636264366166663466356231643065303337636534653f733d393626643d6d6d26723d67) **Ian Buchan** commented on **2017-04-05 20:19:59** Super slick trick. I'm working with AWS API Gateway with a .NET Kestrel server living in a Lambda function that's invoked via the incoming web requests. Every day or so, I would end up with a 404 for my robots.txt file. Since it was only an API, I had no file server to return the robots.txt from. Dynamically generating it solved the problem perfectly! [![RehanSaeed](https://avatars.githubusercontent.com/u/2270364?v=4)](https://github.com/RehanSaeed) ### [RehanSaeed](https://github.com/RehanSaeed) Owner 12 May 2020 [![Wagner Melo](https://camo.githubusercontent.com/d281430d3a1eeeddb4945b84bf105e987cf5082244a4c6556693cb8e81480033/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f35343034623638313736336136663562373031323538366531366565326561643f733d393626643d6d6d26723d67)](https://camo.githubusercontent.com/d281430d3a1eeeddb4945b84bf105e987cf5082244a4c6556693cb8e81480033/68747470733a2f2f7777772e67726176617461722e636f6d2f6176617461722f35343034623638313736336136663562373031323538366531366565326561643f733d393626643d6d6d26723d67) **Wagner Melo** commented on **2020-05-10 00:51:14** Don't forget to turn on attribute based routing in ASP.NET MVC 5 when you register your routes in `RouteConfig`. It's a simple line statement. ``` public static void RegisterRoutes(RouteCollection routes) { routes.IgnoreRoute("{resource}.axd/{*pathInfo}"); routes.MapMvcAttributeRoutes(); // ... } ``` ```json {"@context":"https://schema.org","@graph":[{"@id":"https://rehansaeed.com/#website","@type":"WebSite","description":"Software Developer at Microsoft, YouTuber, Open Source Contributor and Blogger","inLanguage":"en-GB","name":"Muhammad Rehan Saeed","url":"https://rehansaeed.com/","author":{"@id":"https://rehansaeed.com/#identity"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://rehansaeed.com/?search={search_term_string}"},"query-input":"required name=search_term_string"}],"publisher":{"@id":"https://rehansaeed.com/#publisher"}},{"@id":"https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/#webpage","@type":"WebPage","dateModified":"2015-07-31T00:00:00+00:00","datePublished":"2015-07-31T00:00:00+00:00","description":"How to dynamically generate a robots.txt file using a simple ASP.NET MVC action method and only a few lines of code.","name":"Dynamically Generating Robots.txt Using ASP.NET MVC","url":"https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/","breadcrumb":{"@id":"https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/#breadcrumb"},"isPartOf":{"@id":"https://rehansaeed.com/#website"},"potentialAction":[{"@type":"ReadAction","target":["https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/"]}]},{"@id":"https://rehansaeed.com/#identity","@type":"Person","name":"Muhammad Rehan Saeed","url":"https://rehansaeed.com/about/","image":{"@id":"https://rehansaeed.com/#/schema/image/1"},"sameAs":["https://github.com/RehanSaeed","https://www.linkedin.com/in/muhammad-rehan-saeed","https://stackoverflow.com/users/1212017/muhammad-rehan-saeed","https://www.youtube.com/c/MuhammadRehanSaeed","https://www.twitch.tv/rehansaeeduk"]},{"@id":"https://rehansaeed.com/#publisher","@type":"Organization","logo":"https://rehansaeed.com/images/schema/Publisher-600x60.png","name":"Muhammad Rehan Saeed","url":"https://rehansaeed.com"},{"@id":"https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/#article","@type":"Article","dateModified":"2015-07-31T00:00:00+00:00","datePublished":"2015-07-31T00:00:00+00:00","description":"How to dynamically generate a robots.txt file using a simple ASP.NET MVC action method and only a few lines of code.","headline":"Dynamically Generating Robots.txt Using ASP.NET MVC","inLanguage":"en-GB","url":"https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/","author":{"@id":"https://rehansaeed.com/#identity"},"image":[{"@type":"ImageObject","url":"https://rehansaeed.com/images/hero/Robots.txt-1366x768.jpg","caption":"Robots.txt","width":"1366","height":"768"}],"isPartOf":{"@id":"https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/#webpage"},"keywords":[".NET","ASP.NET MVC","ASP.NET Core Boilerplate","C#","Robots","Search Engine Optimization (SEO)","Search Engines","Web Crawlers"],"mainEntityOfPage":{"@id":"https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/#webpage"},"publisher":{"@id":"https://rehansaeed.com/#publisher"}},{"@id":"https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/#breadcrumb","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","name":"Blog","item":"https://rehansaeed.com/","position":1},{"@type":"ListItem","name":"Dynamically Generating Robots.txt Using ASP.NET MVC","item":"https://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/","position":2}]},{"@id":"https://rehansaeed.com/#/schema/image/1","@type":"ImageObject","contentUrl":"https://rehansaeed.com/images/author/Muhammad-Rehan-Saeed/Logo-512x512.png","inLanguage":"en-GB","url":"https://rehansaeed.com/images/author/Muhammad-Rehan-Saeed/Logo-512x512.png"}]} ``` ## Sitemap See the full [sitemap](https://rehansaeed.com/sitemap.md) for all pages.