Serialize Razor code document

[jjonescz]

Resolves #8643.

cc @chsienki

[DustinCampbell]

FWIW, it worries me a bit that we're going further down the rabbit hole of using JSON.NET and lots of JsonConverters. We're actively trying to get away from that in the tooling layers for performance and lower allocation. Is there any reason to use JSON.NET here? Or, could this use System.Text.Json or MessagePack? Or, is the intent to just get things working now and rework it later?

[jjonescz]

We're actively trying to get away from that in the tooling layers for performance and lower allocation.

I wrote this code some time ago and did not know this back then.

is the intent to just get things working now and rework it later?

So I guess this is the current intent.

Is there any reason to use JSON.NET here

Wanted to use the existing TagHelperDescriptor converters. Although now there are more converters in the tooling layer, when I started I'm not sure if they weren't there or I didn't know about them, anyway this PR basically uses some other (legacy) converters from compiler layer. Specifically these ones:

razor/src/Compiler/Microsoft.NET.Sdk.Razor.SourceGenerators/Microsoft.NET.Sdk.Razor.SourceGenerators.csproj

Lines 17 to 21 in ab893b6

	<ItemGroup>
	<Compile Include="..\shared\JsonReaderExtensions.cs" LinkBase="Shared" />
	<Compile Include="..\shared\RazorDiagnosticJsonConverter.cs" LinkBase="Shared" />
	<Compile Include="..\shared\TagHelperDescriptorJsonConverter.cs" LinkBase="Shared" />
	</ItemGroup>

[DustinCampbell]

Wanted to use the existing TagHelperDescriptor converters

FWIW, there is no longer a TagHelperDescriptor converter in the tooling layer. Instead, there are only JsonConverters for "top-level" types -- i.e.., objects that are passed as arguments or results via StreamJsonRpc or serialized directly (like ProjectRazorJson). None of the tooling JsonConverters call into JsonSerializer.Serialize(...) or JsonSerializer.Deserialize(...).

The goal for serialization in the tooling layer is to have an abstraction that we can use (in the near future) to dedupe common object instances and strings in the data, and (again later) switch to a more efficient serialization system. I hope we can just build on this work eventually, though I totally get wanting to just get things working now. This an area that I've actively been working on due to performance analysis that have pointed to our serialization.

[jjonescz]

@dotnet/source-build-internal how do I fix "new prebuilts detected" for Newtonsoft.Json? Is this fine?

[MichaelSimons]

Thanks for asking @jjonescz. Source build must build all dependent binaries from source. External dependencies such as Newtonsoft.Json are contained in the https://github.com/dotnet/source-build-externals repo. To fix this prebuilt, you simply need to add the appropriate version.details.xml dependency to this repo and add a darc subscription to keep it up to date.

    <Dependency Name="Microsoft.SourceBuild.Intermediate.source-build-externals" Version="8.0.0-alpha.1.23312.1">
      <Uri>https://github.com/dotnet/source-build-externals</Uri>
      <Sha>50600f7fcd6e56a496233a859ed7d4d90c173b0d</Sha>
      <SourceBuild RepoName="source-build-externals" ManagedOnly="true" />
    </Dependency>

When razor CI runs the source-build leg, this package will be pulled in by the source-build infrastructure from source-build-externals and therefore will not be considered a prebuilt.

[jjonescz]

That's what I originally tried, but it doesn't get rid of the error. I redid the change, so you can see - it fails currently in CI. Can the reason be that I'm using Newtonsoft.Json version 13.0.3 but there's 13.0.1 in source-build-externals (I think)? But using 13.0.1 explicitly would get NuGet complaining about package downgrade (since there's Newtonsoft.Json 13.0.3 flowing into other projects from Roslyn) - although I guess I could use 13.0.1 for this source-built project and keep 13.0.3 for others if that will solve the problem...