Question: How to use an Azure DevOps Symbol Server inside of PerfView?

[JustinKaffenberger]

It seems for some time now Azure DevOps has supported publishing symbols, and, as far as I can tell, each Azure DevOps organization has a dedicated symbol server.

My question is, how can I, if possible, use this Symbol Server inside of PerfView (for example, using the Set Symbol Path action)?

[brianrob]

From a quick read of the article, it appears that AzDO supports various types of symbol drops, from file shares to indexed symbols. At a minimum you should be able to point at a UNC file path or a symbol server using Set Symbol Path, and that should work. If it doesn't, then the PerfView log should contain information about what went wrong.

[JustinKaffenberger]

That makes sense, I think I'll need to open a separate issue with Microsoft (not sure where yet) to clarify what the actual URI is to the symbol server. My other concern is that its a private symbol server, so I'm not sure how PerfView would behave in that scenario.

[mjsabby]

@JustinKaffenberger At this very moment you would need a proxy server that uses your PAT Token and issues the request on your behalf and you configure that proxy server in PerfView. I'm looking to see if there is a way we could make this setup easier by having PerfView host a proxy server and excavating your AAD credentials from your login, but that is a ways out for me.

In the meantime, however, you could basically spin up a new ASP.NET Core app every time you'd like to debug or what I've done is setup a Windows service that starts it automatically on my dev machine.

I've pasted below all the code you'd need to open up in VS and press F5, then put in http://127.0.0.1:5000/api/symbols/download as your symbol server in PerfView, or you could set _NT_SYMBOL_PATH so it is even more seamless.

The code below expects your PAT Token in the environment variable "PATToken". Of course follow all the guidelines of creating scoped PAT tokens, etc. and note that saving them in environment variables registry is not considered secure, yada yada ...

Here is AADSymbolProxy.csproj

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

</Project>

Here is Program.cs

namespace AADSymbolProxy
{
    using System;
    using System.Collections.Generic;
    using System.Net;
    using System.Net.Http;
    using System.Text;
    using System.Threading.Tasks;
    using Microsoft.AspNetCore.Builder;
    using Microsoft.AspNetCore.Hosting;
    using Microsoft.AspNetCore.Mvc;
    using Microsoft.Extensions.DependencyInjection;
    using Microsoft.Extensions.Hosting;

    public static class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) => Host.CreateDefaultBuilder(args).ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); });
    }

    public sealed class Startup
    {
        public void ConfigureServices(IServiceCollection services)
        {
            var list = new Dictionary<string, string>
            {
                {
                    "FIXME", "https://FIXME__YOUR__ACCOUNT.artifacts.visualstudio.com/defaultcollection/_apis/symbol/symsrv/"
                },
                {
                    "msdl", "https://msdl.microsoft.com/download/symbols/"
                },
                {
                    "nuget", "https://symbols.nuget.org/download/symbols/"
                }
            };

            services.AddSingleton(list);

            var token = Environment.GetEnvironmentVariable("PATToken"); // or KeyVault, etc.
            var base64PatString = $"Basic {Convert.ToBase64String(Encoding.ASCII.GetBytes($"{string.Empty}:{token}"))}";

            foreach (var item in list)
            {
                services.AddHttpClient(item.Key, configureClient =>
                {
                    configureClient.BaseAddress = new Uri(item.Value);
                    configureClient.DefaultRequestHeaders.Add("Authorization", base64PatString);
                    configureClient.DefaultRequestHeaders.Add("SymbolChecksumValidationSupported", "1");
                }).ConfigurePrimaryHttpMessageHandler(provider => new SocketsHttpHandler { AllowAutoRedirect = false, UseCookies = false });
            }

            services.AddControllers();
        }

        public void Configure(IApplicationBuilder app)
        {
            app.UseRouting();
            app.UseEndpoints(endpoints => endpoints.MapControllers());
        }
    }

    [Route("api/[controller]")]
    [ApiController]
    public sealed class SymbolsController : ControllerBase
    {
        private readonly IHttpClientFactory clientFactory;

        private readonly Dictionary<string, string> dict;

        public SymbolsController(IHttpClientFactory clientFactory, Dictionary<string, string> dict)
        {
            this.clientFactory = clientFactory;
            this.dict = dict;
        }

        [HttpGet]
        [Route("download/{filename}/{key}/{filename2}")]
        public async Task<ActionResult> Symbols(string filename, string key, string filename2)
        {
            foreach (var item in this.dict)
            {
                var server = item.Key;
                var client = clientFactory.CreateClient(server);

                using (HttpResponseMessage response = await client.GetAsync(filename + "/" + key + "/" + filename2, HttpCompletionOption.ResponseHeadersRead))
                {
                    if (response.StatusCode == HttpStatusCode.NotFound)
                    {
                        continue;
                    }

                    if (response.StatusCode == HttpStatusCode.Unauthorized)
                    {
                        continue;
                    }

                    if (response.StatusCode == HttpStatusCode.Found && response.Headers.Location != null)
                    {
                        return new RedirectResult(response.Headers.Location.AbsoluteUri);
                    }
                }
            }

            return new NotFoundResult();
        }
    }
}

[JustinKaffenberger]

@mjsabby thanks for the thorough workaround. I think I'm still hung up on the URL needed for the new Azure DevOps services, as I'm pretty sure they differ from the old VSTS URLs. This is not well documented on the Azure DevOps side of things, so I will close this issue for now.

[mjsabby]

@JustinKaffenberger so authentication is not your problem?

https://FIXME__YOUR__ACCOUNT.artifacts.visualstudio.com/defaultcollection/_apis/symbol/symsrv/

is your symbol server url.

[buckleyGI]

That's genius @mjsabby, the idea of a proxy, and PerfView now fetches my pdb's from DevOps!

However, did you also get PerfView to display the source code? Or was your goal to only get the PDB?

My pdb contains the exact command: tf.exe with, among others, the commit id.
This is all done for us when using DevOps. So all the information to download the source code seems at hand.

But is PerfView now also supposed to connect DevOps for the source code?
I don't see in the log of PerfView a command to fetch the source code. It is as if the tf.exe command is ignored.

Or is the way you now proceed to download the source file by hand and place it in a path you can set with _NT_SOURCE_PATH ?

Thanks again for the proxy code.

[pharring]

Consider using SourceLink. It's the new source server. https://aka.ms/sourcelink

[buckleyGI]

@pharring Isn't SourceLink exclusively for nuget packages?
The sources we have in DevOps are for our internal products. We don't develop any libraries that we distribute to the general public.

Is SourceLink also a solution here? Are you proposing to use it a bit outside of its intended use?

[buckleyGI]

The end result I am aiming for is that PerfView can download the source as well.
I think that enabling SourceLink won't enable this for the existing PerfView as well.

Is this kind of seamless integration with DevOps (symbol + source download) a scenario that PerfView is aiming to support?
Or are there some manual steps that need to be taken by the user. Clarifying this will make clear for me of what to expect here and not trying something that won't work at all while I get to know this side of PerfView.

Is it right to say that we can with a little modification to PerfView enable the auto download of the pdb but that source code downloading isn't as straightforward and isn't planned to be supported?

[pharring]

SourceLink is not just for nuget packages. It is the recommended alternative to source server. The problem with source server is that it allows for arbitrary command execution which, as you can imagine, is vulnerable to abuse.

You'll still need a proxy to download PDBs from authenticated symbol servers.

PerfView knows how to decode SourceLink information and "go to source" will attempt to download source using the SourceLink info. If your source code is on a private source repo, you'll need a similar proxy solution to supply credentials.

[pharring]

We should be able to eliminate the need for a local proxy and make "go to source" seamless by leveraging Visual Studio or VS Code sign-in credentials. You would need to have VS Code or Visual Studio installed and signed in (to your AzDO instance) on the same machine where you're running PerfView.

See also #1563 which proposes adding a sign-in experience to PerfView itself.

[Xhanti]

The only caveat to source link is private gitlab instances are not supported. Currently a proxy in the middle to bridge the gap is the only way around this

[pharring]

GitLab is supported: https://www.nuget.org/packages/Microsoft.SourceLink.GitLab

[Xhanti]

Hi @pharring ,

Thank you for the feedback.
Last I checked, this was the issue I ran into: dotnet/sourcelink#281
It looks GCM may have solved some of these issues, I'll have to play around with it in VS and perfview.

Regards

[pharring]

PerfView doesn't use GCM -- but it could.

[pharring]

I have a working branch with Git Credential Manager authentication added. It needs some UI work, but it should be ready for PR tomorrow. Right now, it supports GitHub only, but that's mainly because I don't have ready access to a private GitLab instance to test it.

[Xhanti]

When its merged in, I could clone the repo and see if it works for the private instance I have access to and report any findings I have

[pharring]

@Xhanti, it's merged to main now. There's a TODO in the code to add GitLab support:
https://github.com/microsoft/perfview/blob/main/src/PerfView/SymbolReaderHttpHandler.cs#L795

[Xhanti]

@pharring thank you for the update. I'll take this for a spin and post here if it works 🙂