Enable maintenance-packages in source build #140

ericstj · 2024-10-21T22:04:36Z

This enables source-build pass on maintenance-packages. dotnet/source-build#4684

If there are any blockers with this we could resort back to conditioning the uptake of these packages so that SB would only see versions that are available in SBRP, but I want to try to do the better thing first.

With this change I still see the following prebuilts that I would like help resolving:

: error : 8 new pre-builts discovered! Detailed usage report can be found at /workspaces/maintenance-packages/artifacts/source-build/self/prebuilt-report/baseline-comparison.xml.
: error : See https://aka.ms/dotnet/prebuilts for guidance on what pre-builts are and how to eliminate them.
: error : Package IDs are:
: error : Microsoft.AspNetCore.App.Ref.6.0.35
: error : Microsoft.NET.Sdk.IL.8.0.0
: error : Microsoft.NETCore.App.Ref.6.0.35
: error : Microsoft.NETFramework.ReferenceAssemblies.1.0.3
: error : Microsoft.NETFramework.ReferenceAssemblies.net462.1.0.3
: error : Microsoft.NETFramework.ReferenceAssemblies.net472.1.0.3
: error : runtime.linux-x64.Microsoft.NETCore.ILAsm.6.0.0
: error : runtime.linux-x64.Microsoft.NETCore.ILDAsm.6.0.0

The ref-packs are a common problem that everyone should hit, maybe we're just missing something. For IL, we need a way to consume those outputs of runtime as a toolset dependency. I'd like some help from @dotnet/source-build to recommend how to fix these.

NikolaMilosavljevic · 2024-10-21T22:22:02Z

Are your projects conditioned to not target .NET FX in source build? That might be the reason for some of the prebuilts.

eng/SourceBuilld.props

MichaelSimons · 2024-10-21T22:44:12Z

A source-build leg should be added as part of this PR. Should be easy to set the enableSourceBuild parameter on the jobs template. A side effect of this is it would easily allow me to see the prebuilt report to know where the prebuilts are coming from.

MichaelSimons · 2024-10-21T22:48:24Z

I was expecting that you would need a Version.Details.xml ProductDependency on source-build-reference-packages.

eng/pipelines/maintenance-packages-pr.yml

MichaelSimons · 2024-10-22T14:41:17Z

I was expecting that you would need a Version.Details.xml ProductDependency on source-build-reference-packages.

I see this is defined as a ToolsetDependency. I don't think that matters that it is a toolset vs product dependency.

I see arcade and the SBRP references are 8.0. I assume this repo is a version agnostic repo and will be referenced by multiple versions of the product? This may present some challenges if we make breaking changes to the arcade/UB/SB infra over time.

MichaelSimons · 2024-10-22T14:52:31Z

Are your projects conditioned to not target .NET FX in source build? That might be the reason for some of the prebuilts.

I took a brief look and confirmed this is the case. Source build must exclude .NET FX TFMs.

ericstj · 2024-10-22T16:34:24Z

@NikolaMilosavljevic / @MichaelSimons - what about getting the ILProj SDK and ILDASM/ILASM? These are built by runtime. I'd want to tell source build that, and let it upgrade them with the built copy on the second pass.

We exclude NetFx from packages for source build too? I didn't realize that -- I thought we'd still build that stuff since it's sprinkled throughout the product (eg: task folders). If we're actually excluding netfx then I wonder if we really want to bring this repo into source build. Most of the runtime assemblies it provides are only for older frameworks - so source build should be fine with just reference surface area. Then again - we'd need to continuously regenerate that surface area as we build new package. @ViktorHofer what do you think?

…packages into sourceBuild

ericstj · 2024-10-22T16:47:51Z

eng/pipelines/maintenance-packages.yml

@@ -67,6 +67,7 @@ extends:
          enableMicrobuild: true
          enablePublishUsingPipelines: true
          publishAssetsImmediately: true
+          enableSourceBuild: true


@MichaelSimons Do we require source-build be enabled for official builds? I hadn't set this originally in my change.

Yes - It is required in order to produce the SB intermediate package that downstream repos would pull in during their source-build legs. This is what eliminates having to add everything as a "known/allowed" prebuilt.

ViktorHofer · 2024-10-22T16:53:49Z

Yes, the .NET product, built from source doesn't support building .NET Framework assemblies anymore. That was a pretty big effort that Michael's team drove in the last two (?) years to reduce the source-build artifact size and build times. The .NET Framework targeting packs also got removed from SBRP.

@ViktorHofer what do you think?

We already need to add new versions of actively developed packages to SBRP, i.e. STJ: https://github.com/dotnet/source-build-reference-packages/tree/main/src/referencePackages/src/system.text.json

IIRC the plan for maintenance-packages is to only release new package versions on demand (or when a TFM that is targeted goes out-of-support). That should be way less often than its current traffic, maybe only once very two years.

To onboard this repository to the source-build graph, it needs to be added to the VMR which I would rather avoid. Honestly, it feels wrong to me to add a maintenance-only repository to it.

MichaelSimons · 2024-10-22T17:07:43Z

IIRC the plan for maintenance-packages is to only release new package versions on demand (or when a TFM that is targeted goes out-of-support). That should be way less often than its current traffic, maybe only once very two years.

To onboard this repository to the source-build graph, it needs to be added to the VMR which I would rather avoid. Honestly, it feels wrong to me to add a maintenance-only repository to it.

Onboarding maintenance-packages arose as a result of this runtime PR. The issue source build has is the dependency on the preview versions. Help me understand why this is versus the latest release of the packages? This is what causes the problem with source-build. Adding the preview versions to SBRP is not scalable in general and is against the current policy.

Note: All new packages should be for released stable versions. Adding preview/release candidate packages are for exceptional cases only and require approval from dotnet/source-build.

ericstj · 2024-10-22T20:12:21Z

These packages binaries do not ship inside the source-built product. They are needed when building intermediates that might target netstandard2.0 or older versions of .NET. The fact that we work today with SBRP is proof of this.

maintenance-packages itself is a special repo that lives outside the product. It's not part of 8.0, 9.0, or 10.0 -- it's on its own where things go after they're not part of the latest .NET. I can see it remaining separate. It's also going to lag behind the latest product's infrastructure since it's a servicing repo that's meant to ship at any time, it will stay on the LTS infra. I can see this being a problem when it needs to be consumed by the latest in development release if we try to force it into the product. It was really an open question to me if it belonged in source-build or not.

We don't intend to have the product take a dependency on pre-release versions moving forward. We're just doing it this first time to test and understand if we need to do anything before the first stable release of maintenance-packages. In the future releases from m-p will look a lot like servicing releases from a previous servicing band. IIUC those are handled by SBRP today. @MichaelSimons does this address your questions?

MichaelSimons · 2024-10-24T15:54:58Z

We don't intend to have the product take a dependency on pre-release versions moving forward. We're just doing it this first time to test and understand if we need to do anything before the first stable release of maintenance-packages. In the future releases from m-p will look a lot like servicing releases from a previous servicing band. IIUC those are handled by SBRP today. @MichaelSimons does this address your questions?

Yes that answer's my question. I would be amenable to adding the pre-release versions to SBRP for this initial setup with the understanding that we would clean them up/upgrade once we have the first stable release of maintenance-packages. This would eliminate the need to SB maintenance-package/include them in the VMR.

carlossanlop · 2024-10-28T17:14:28Z

.github/CODEOWNERS

+# Users referenced in this file will automatically be requested as reviewers for PRs that modify the given paths.
+# See https://help.github.com/articles/about-code-owners/
+
+/eng/SourceBuild*                                                                 @dotnet/source-build


This is a strange error, @dotnet/source-build does exist:

Unknown owner on line 4: make sure the team @dotnet/source-build exists, is publicly visible, and has write access to the repository

I believe that the team has to have write access.

carlossanlop · 2024-10-28T17:29:18Z

Directory.Packages.props

@@ -19,7 +19,7 @@
    <PackageVersion Include="System.ValueTuple" Version="4.5.0" />
  </ItemGroup>

-  <ItemGroup>
+  <ItemGroup Condition="'$(DotNetBuildFromSource)' != 'true'">
    <GlobalPackageReference Include="Microsoft.DotNet.ApiCompat.Task" Version="9.0.100-preview.2.24102.10" />


There's a git merge conflict with this line. In main, the version is 9.0.100-rc.2.24474.11.

carlossanlop · 2024-10-28T17:29:52Z

Directory.Build.props

+    <!-- maintenance-packages doesn't support Arcade-driven target framework filtering. -->
+    <NoTargetFrameworkFiltering>true</NoTargetFrameworkFiltering>


There's a git merge conflict with these lines. In main, these lines have:

 <WarningsNotAsErrors Condition="'$(OfficialBuild)' != 'true'">$(WarningsNotAsErrors);NU1901;NU1902;NU1903;NU1904</WarningsNotAsErrors>

carlossanlop · 2024-10-28T21:17:05Z

@MichaelSimons I am of the opinion that maintenance-packages should be excluded from source-build as the assemblies offered here originally were migrated from the previous .NET versions that are now out of support. They don't belong to .NET Latest.

What course of acftion would you recommend to do that, aside from closing this PR?

MichaelSimons · 2024-10-29T14:09:48Z

What course of action would you recommend to do that, aside from closing this PR?

This is what I had in mind. I would like to know a rough time period for item 4. Is that a few weeks or not until 10 ships?

Establish an agreement that after we get past the initial release of maintenance packages (e.g. enable stable versioning), that only stable versions of the maintenance packages will be reference by the runtime.
Close this PR
Add the preview packages being referenced in Bump versions of maintenance-packages dependencies consumed in runtime runtime#108806 to SBRP
Work through the initial release of maintenance packages to the point where stable versioning is enabled.
Flow the stable versions to runtime which will require adding the stable versioned packages to SBRP first.

ericstj · 2024-10-29T17:18:09Z

Sounds like a good plan, thank you @MichaelSimons.

carlossanlop · 2024-10-29T17:22:01Z

Thank you @ericstj and @MichaelSimons

ericstj added 3 commits October 21, 2024 14:16

Add source build files

66015f3

Fix source build issues

576f9f6

Enable source build during PR validation and rolling builds

6c95bdc

ericstj requested review from carlossanlop, ViktorHofer and a team October 21, 2024 22:04

MichaelSimons reviewed Oct 21, 2024

View reviewed changes

eng/SourceBuilld.props Outdated Show resolved Hide resolved

Enable source-build leg in PR/CI

6912544

MichaelSimons reviewed Oct 22, 2024

View reviewed changes

eng/pipelines/maintenance-packages-pr.yml Show resolved Hide resolved

Merge branch 'sourceBuild' of https://github.com/ericstj/maintenance-…

e4af77d

…packages into sourceBuild

ericstj commented Oct 22, 2024

View reviewed changes

carlossanlop reviewed Oct 28, 2024

View reviewed changes

ericstj closed this Oct 29, 2024

carlossanlop mentioned this pull request Oct 29, 2024

Update runtime dependency packages to latest from maintenance-packages dotnet/source-build-reference-packages#1068

Merged

MichaelSimons mentioned this pull request Oct 31, 2024

Onboard maintenance-packages to source-build dotnet/source-build#4684

Closed

MichaelSimons mentioned this pull request Nov 21, 2024

Bump versions of maintenance-packages dependencies consumed in runtime dotnet/runtime#108806

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable maintenance-packages in source build #140

Enable maintenance-packages in source build #140

ericstj commented Oct 21, 2024 •

edited

Loading

NikolaMilosavljevic commented Oct 21, 2024

MichaelSimons commented Oct 21, 2024

MichaelSimons commented Oct 21, 2024

MichaelSimons commented Oct 22, 2024

MichaelSimons commented Oct 22, 2024

ericstj commented Oct 22, 2024

ericstj Oct 22, 2024

MichaelSimons Oct 22, 2024

ViktorHofer commented Oct 22, 2024

MichaelSimons commented Oct 22, 2024

ericstj commented Oct 22, 2024 •

edited

Loading

MichaelSimons commented Oct 24, 2024

carlossanlop Oct 28, 2024

ellahathaway Oct 28, 2024

carlossanlop Oct 28, 2024

carlossanlop Oct 28, 2024

carlossanlop commented Oct 28, 2024

MichaelSimons commented Oct 29, 2024

ericstj commented Oct 29, 2024

carlossanlop commented Oct 29, 2024

		<!-- maintenance-packages doesn't support Arcade-driven target framework filtering. -->
		<NoTargetFrameworkFiltering>true</NoTargetFrameworkFiltering>

Enable maintenance-packages in source build #140

Enable maintenance-packages in source build #140

Conversation

ericstj commented Oct 21, 2024 • edited Loading

NikolaMilosavljevic commented Oct 21, 2024

MichaelSimons commented Oct 21, 2024

MichaelSimons commented Oct 21, 2024

MichaelSimons commented Oct 22, 2024

MichaelSimons commented Oct 22, 2024

ericstj commented Oct 22, 2024

ericstj Oct 22, 2024

Choose a reason for hiding this comment

MichaelSimons Oct 22, 2024

Choose a reason for hiding this comment

ViktorHofer commented Oct 22, 2024

MichaelSimons commented Oct 22, 2024

ericstj commented Oct 22, 2024 • edited Loading

MichaelSimons commented Oct 24, 2024

carlossanlop Oct 28, 2024

Choose a reason for hiding this comment

ellahathaway Oct 28, 2024

Choose a reason for hiding this comment

carlossanlop Oct 28, 2024

Choose a reason for hiding this comment

carlossanlop Oct 28, 2024

Choose a reason for hiding this comment

carlossanlop commented Oct 28, 2024

MichaelSimons commented Oct 29, 2024

ericstj commented Oct 29, 2024

carlossanlop commented Oct 29, 2024

ericstj commented Oct 21, 2024 •

edited

Loading

ericstj commented Oct 22, 2024 •

edited

Loading