Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression: ANCM: Unexpected application shutdown #58939

Open
1 task done
matbech opened this issue Nov 14, 2024 · 6 comments
Open
1 task done

Regression: ANCM: Unexpected application shutdown #58939

matbech opened this issue Nov 14, 2024 · 6 comments
Labels
area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions feature-iis Includes: IIS, ANCM Needs: Attention 👋 This issue needs the attention of a contributor, typically because the OP has provided an update.

Comments

@matbech
Copy link

matbech commented Nov 14, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Describe the bug

Starting with .NET 9 Preview 4 up to the current version of .NET 9 (9.0.100 with ANCM 19.0.24303.0) we are seeing unexpected application shutdowns (logged as "Application 'MACHINE/WEBROOT/APPHOST/WWW.BLA.COM/APPPOOLNAME' has shutdown.") after a certain time (usually a couple of hours). This results in the application being non-functional (503 http errors) until the IIS application pool is restarted or recycled.

The IIS application pool is configured with the following settings:
Idle time-out: 0
StartMode: AlwaysRunning.
Managed Pipeline mode: Integrated
Rapid Fail Protection - Enabled: false

OS: Windows Server 2022 (x64) and Windows Server 2025 (x64)

We have compiled the application with both, the .NET 9 Preview 3 SDK and with .NET 9.0.100 SDK with the same result when running it under a runtime / ANCM > .NET 9 Preview 3.

The issue does not happen with .NET 9 Preview 3 (runtime and ANCM). Maybe related to this issue #41340 and this commit #52807

No additional errors or informational messages (e.g. no application shutdowns due to recycle), or crash reports (WER) can be found in the event log.

The application is trivial. For illustration purpose, the simplified source code follows:

Program.cs

namespace Service
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
             .ConfigureWebHostDefaults(webBuilder =>
             {
                 webBuilder.UseStartup<Startup>();
             });
    }
}

Startup.cs


namespace Service
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMvc(options =>
            {
                options.OutputFormatters.Add(new XmlDataContractSerializerOutputFormatter());
                options.InputFormatters.Add(new XmlDataContractSerializerInputFormatter(options));
            });
        }

        public void Configure(IApplicationBuilder app, IHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                // managed by IIS
                // app.UseHsts();
            }

            // managed by IIS
            //app.UseHttpsRedirection();
            app.UseRouting();
            app.UseEndpoints(route =>
            {
                route.MapControllers();
            });
        }
    }
}

ActivationController.cs


namespace Service.Controllers
{
    [Route("v1")]
    [ApiController]
    public class ActivationController : ControllerBase, IDisposable
    {
        public ActivationController(ILogger<ActivationController> logger, IConfiguration configuration)
        {
            _logger = logger;
            _configuration = configuration;
        }
 
        public void Dispose()
        {
           // cleanup of db
        }

        [HttpPost]
        [Produces("application/xml")]
        public Business.Model.ActivationResponse Activate(Business.Model.ActivationRequest request)
        {
           ....
            return response;
        }

        [Route("deactivate")]
        [HttpPost]
        [Produces("application/xml")]
        public Business.Model.DeactivationResponse Deactivate(Business.Model.DeactivationRequest request)
        {
            ...
            return response;
        }

        private readonly ILogger _logger;
        private readonly IConfiguration _configuration;
        private MySqlConnection _licenseDatabase;
    }
}

Expected Behavior

No shutdown of the application. Application stays alive.

Steps To Reproduce

I'm only able to reproduce this in a production environment and it takes a random interval (hours) until it happens.

Exceptions (if any)

No response

.NET Version

9.0.100

Anything else?

dotnet --info

Host:
Version: 9.0.0
Architecture: x64
Commit: 9d5a6a9aa4
RID: win-x64

.NET SDKs installed:
No SDKs were found.

.NET runtimes installed:
Microsoft.AspNetCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 9.0.0-preview.3.24172.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
Not found
@dotnet-issue-labeler dotnet-issue-labeler bot added the area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions label Nov 14, 2024
@gfoidl gfoidl added the feature-iis Includes: IIS, ANCM label Nov 14, 2024
@BrennanConroy
Copy link
Member

Share logs please:
https://learn.microsoft.com/aspnet/core/host-and-deploy/iis/logging-and-diagnostics?view=aspnetcore-9.0#enhanced-diagnostic-logs

@BrennanConroy BrennanConroy added the Needs: Author Feedback The author of this issue needs to respond in order for us to continue investigating this issue. label Nov 20, 2024
@ZyuanC
Copy link

ZyuanC commented Nov 21, 2024 •
edited
Loading

I have the same issue with my application.

Description

I have two websites, ASP.NET Core API and WordPress, in the same application pool.
Before, I used the dotnet 8.0.7 hosting bundle, and it worked fine.
After installing the dotnet 9.0.0 hosting bundle, whenever an HTTP Error 500 is triggered by the WordPress site, it forcefully kills all dotnet subprocesses of the w3wp.exe process. As a result, the ASP.NET Core API no longer starts and returns an HTTP Error 503, service is unavailable. If I fix the WordPress HTTP Error 500, the WordPress will work again, however, I have to restart the application pool to make the ASP.NET Core API work.

Before WordPress triggers an HTTP Error 500

OutOfProcess hosting model

The processes:

Handles      WS(K)   CPU(s)     Id UserName  ProcessName
-------      -----   ------     -- --------  -----------
     87      11288     0.02  12276 henryxxx  conhost
    463      96464     2.69  32100 henryxxx  dotnet
    490      22120     0.31  67660 henryxxx  w3wp

The aspnetcore-debug log:

[2024-11-21T05:05:57.210Z, PID: 67660] [aspnetcorev2.dll] Initializing logs for 'C:\Program Files (x86)\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 67660. File Version: 19.0.24303.0. Description: IIS ASP.NET Core Module V2. Commit: .
[2024-11-21T05:05:57.230Z, PID: 67660] [aspnetcorev2.dll] Loading request handler:  'C:\Program Files (x86)\IIS\Asp.Net Core Module\V2\19.0.24303\aspnetcorev2_outofprocess.dll'
[2024-11-21T05:05:57.254Z, PID: 67660] [aspnetcorev2.dll] Creating handler application
[2024-11-21T05:05:57.278Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Initializing logs for 'C:\Program Files (x86)\IIS\Asp.Net Core Module\V2\19.0.24303\aspnetcorev2_outofprocess.dll'. Process Id: 67660. File Version: 19.0.24303.0. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: .
[2024-11-21T05:05:57.289Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Starting app_offline monitoring in application 'h:\root\home\henryxxx\www\homenew\'
[2024-11-21T05:05:57.300Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Starting file watcher thread
[2024-11-21T05:05:57.300Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::FORWARDING_HANDLER
[2024-11-21T05:05:59.981Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Event Log: 'Application '/LM/W3SVC/1074629900/ROOT' started process '32100' successfully and process '32100' is listening on port '26945'.' 
End Event Log Message.
[2024-11-21T05:06:00.002Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnWinHttpCompletionInternal 10 -- 87416 --0594F638

[2024-11-21T05:06:00.013Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnWinHttpCompletionInternal 20 -- 87416 --0594F638

[2024-11-21T05:06:00.303Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnWinHttpCompletionInternal 400000 -- 87416 --0594F638

[2024-11-21T05:06:00.313Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnWinHttpCompletionInternal 20000 -- 87416 --0594F638

[2024-11-21T05:06:00.322Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnWinHttpCompletionInternal 40000 -- 19280 --0594F638

[2024-11-21T05:06:00.331Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnWinHttpCompletionInternal 80000 -- 19280 --0594F638

[2024-11-21T05:06:00.341Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnAsyncCompletion Done 1
[2024-11-21T05:06:00.351Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnWinHttpCompletionInternal 40000 -- 15180 --0594F638

[2024-11-21T05:06:00.360Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnWinHttpCompletionInternal 800 -- 15180 --0594F638

[2024-11-21T05:06:00.372Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnAsyncCompletion Done 1
[2024-11-21T05:06:00.382Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::OnAsyncCompletion Done 0
[2024-11-21T05:06:00.392Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::~FORWARDING_HANDLER
[2024-11-21T05:07:38.113Z, PID: 67660] [aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::FORWARDING_HANDLER

InProcess hosting model

Handles      WS(K)   CPU(s)     Id UserName  ProcessName
-------      -----   ------     -- --------  -----------
    142      13804     0.05  77200 henryxxx  conhost
    770      84756     2.41   6288 henryxxx  w3wp

[2024-11-21T05:49:22.479Z, PID: 67660] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalStopListening
[2024-11-21T05:49:23.614Z, PID: 67660] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::Terminate
[2024-11-21T05:49:37.109Z, PID: 6288] [aspnetcorev2.dll] Initializing logs for 'C:\Program Files (x86)\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 6288. File Version: 19.0.24303.0. Description: IIS ASP.NET Core Module V2. Commit: .
[2024-11-21T05:49:37.135Z, PID: 6288] [aspnetcorev2.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\sm.web.tool.dll' path: 'h:\root\home\henryxxx\www\homenew\'
[2024-11-21T05:49:37.152Z, PID: 6288] [aspnetcorev2.dll] Known dotnet.exe location: ''
[2024-11-21T05:49:37.170Z, PID: 6288] [aspnetcorev2.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[2024-11-21T05:49:37.186Z, PID: 6288] [aspnetcorev2.dll] get_hostfxr_path skipped due to expandedProcessPath being dotnet.exe
[2024-11-21T05:49:37.205Z, PID: 6288] [aspnetcorev2.dll] Resolving absolute path to dotnet.exe from 'dotnet.exe'
[2024-11-21T05:49:37.221Z, PID: 6288] [aspnetcorev2.dll] Invoking where.exe to find dotnet.exe
[2024-11-21T05:49:37.327Z, PID: 6288] [aspnetcorev2.dll] where.exe invocation returned: 'C:\Program Files\dotnet\dotnet.exe
C:\Program Files (x86)\dotnet\dotnet.exe
'
[2024-11-21T05:49:37.342Z, PID: 6288] [aspnetcorev2.dll] Current process bitness type detected as isX64=0
[2024-11-21T05:49:37.359Z, PID: 6288] [aspnetcorev2.dll] Processing entry 'C:\Program Files\dotnet\dotnet.exe'
[2024-11-21T05:49:37.375Z, PID: 6288] [aspnetcorev2.dll] Binary type 6
[2024-11-21T05:49:37.389Z, PID: 6288] [aspnetcorev2.dll] Processing entry 'C:\Program Files (x86)\dotnet\dotnet.exe'
[2024-11-21T05:49:37.405Z, PID: 6288] [aspnetcorev2.dll] Binary type 0
[2024-11-21T05:49:37.420Z, PID: 6288] [aspnetcorev2.dll] Found dotnet.exe via where.exe invocation at 'C:\Program Files (x86)\dotnet\dotnet.exe'
[2024-11-21T05:49:37.439Z, PID: 6288] [aspnetcorev2.dll] Trying get_hostfxr_path with dotnet path as dotnet root
[2024-11-21T05:49:37.456Z, PID: 6288] [aspnetcorev2.dll] hostfxr.dotnet_root: 'C:\Program Files (x86)\dotnet'
[2024-11-21T05:49:37.471Z, PID: 6288] [aspnetcorev2.dll] hostfxr.assembly_path: 'h:\root\home\henryxxx\www\homenew\'
[2024-11-21T05:49:37.487Z, PID: 6288] [aspnetcorev2.dll] dotnetExePath 'C:\Program Files (x86)\dotnet\dotnet.exe'
[2024-11-21T05:49:37.502Z, PID: 6288] [aspnetcorev2.dll] hostFxrDllpath 'C:\Program Files (x86)\dotnet\host\fxr\9.0.0\hostfxr.dll'
[2024-11-21T05:49:37.519Z, PID: 6288] [aspnetcorev2.dll] Converted argument '.\sm.web.tool.dll' to 'h:\root\home\henryxxx\www\homenew\.\sm.web.tool.dll'
[2024-11-21T05:49:37.533Z, PID: 6288] [aspnetcorev2.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files (x86)\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files (x86)\dotnet\host\fxr\9.0.0\hostfxr.dll' arguments:
[2024-11-21T05:49:37.549Z, PID: 6288] [aspnetcorev2.dll] Argument[0] = 'C:\Program Files (x86)\dotnet\dotnet.exe'
[2024-11-21T05:49:37.571Z, PID: 6288] [aspnetcorev2.dll] Argument[1] = 'h:\root\home\henryxxx\www\homenew\.\sm.web.tool.dll'
[2024-11-21T05:49:37.587Z, PID: 6288] [aspnetcorev2.dll] Loading hostfxr from location C:\Program Files (x86)\dotnet\host\fxr\9.0.0\hostfxr.dll
[2024-11-21T05:49:37.656Z, PID: 6288] [aspnetcorev2.dll] Canceling standard stream pipe reader
[2024-11-21T05:49:37.731Z, PID: 6288] [aspnetcorev2.dll] Loading request handler:  'C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App\8.0.7\aspnetcorev2_inprocess.dll'
[2024-11-21T05:49:37.748Z, PID: 6288] [aspnetcorev2.dll] Creating handler application
[2024-11-21T05:49:37.759Z, PID: 6288] [aspnetcorev2_inprocess.dll] Initializing logs for 'C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App\8.0.7\aspnetcorev2_inprocess.dll'. Process Id: 6288. File Version: 18.0.24166.7. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: .
[2024-11-21T05:49:37.766Z, PID: 6288] [aspnetcorev2_inprocess.dll] Waiting for initialization
[2024-11-21T05:49:37.774Z, PID: 6288] [aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'h:\root\home\henryxxx\www\homenew\'
[2024-11-21T05:49:37.785Z, PID: 6288] [aspnetcorev2_inprocess.dll] Starting file watcher thread
[2024-11-21T05:49:37.785Z, PID: 6288] [aspnetcorev2_inprocess.dll] Starting in-process worker thread
[2024-11-21T05:49:37.809Z, PID: 6288] [aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\sm.web.tool.dll' path: 'h:\root\home\henryxxx\www\homenew\'
[2024-11-21T05:49:37.821Z, PID: 6288] [aspnetcorev2_inprocess.dll] Known dotnet.exe location: 'C:\Program Files (x86)\dotnet\dotnet.exe'
[2024-11-21T05:49:37.830Z, PID: 6288] [aspnetcorev2_inprocess.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[2024-11-21T05:49:37.837Z, PID: 6288] [aspnetcorev2_inprocess.dll] Trying get_hostfxr_path with dotnet path as dotnet root
[2024-11-21T05:49:37.845Z, PID: 6288] [aspnetcorev2_inprocess.dll] hostfxr.dotnet_root: 'C:\Program Files (x86)\dotnet'
[2024-11-21T05:49:37.852Z, PID: 6288] [aspnetcorev2_inprocess.dll] hostfxr.assembly_path: 'h:\root\home\henryxxx\www\homenew\'
[2024-11-21T05:49:37.862Z, PID: 6288] [aspnetcorev2_inprocess.dll] dotnetExePath 'C:\Program Files (x86)\dotnet\dotnet.exe'
[2024-11-21T05:49:37.870Z, PID: 6288] [aspnetcorev2_inprocess.dll] hostFxrDllpath 'C:\Program Files (x86)\dotnet\host\fxr\9.0.0\hostfxr.dll'
[2024-11-21T05:49:37.880Z, PID: 6288] [aspnetcorev2_inprocess.dll] Converted argument '.\sm.web.tool.dll' to 'h:\root\home\henryxxx\www\homenew\.\sm.web.tool.dll'
[2024-11-21T05:49:37.887Z, PID: 6288] [aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files (x86)\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files (x86)\dotnet\host\fxr\9.0.0\hostfxr.dll' arguments:
[2024-11-21T05:49:37.896Z, PID: 6288] [aspnetcorev2_inprocess.dll] Argument[0] = 'C:\Program Files (x86)\dotnet\dotnet.exe'
[2024-11-21T05:49:37.904Z, PID: 6288] [aspnetcorev2_inprocess.dll] Argument[1] = 'h:\root\home\henryxxx\www\homenew\.\sm.web.tool.dll'
[2024-11-21T05:49:37.912Z, PID: 6288] [aspnetcorev2_inprocess.dll] Setting environment variable ASPNETCORE_IIS_HTTPAUTH=anonymous;
[2024-11-21T05:49:37.922Z, PID: 6288] [aspnetcorev2_inprocess.dll] Setting environment variable ASPNETCORE_IIS_PHYSICAL_PATH=h:\root\home\henryxxx\www\homenew\
[2024-11-21T05:49:37.931Z, PID: 6288] [aspnetcorev2_inprocess.dll] Loading hostfxr from location C:\Program Files (x86)\dotnet\host\fxr\9.0.0\hostfxr.dll
[2024-11-21T05:49:37.938Z, PID: 6288] [aspnetcorev2_inprocess.dll] Initial Dll directory: '', current directory: 'C:\Windows\SysWOW64\inetsrv'
[2024-11-21T05:49:37.947Z, PID: 6288] [aspnetcorev2_inprocess.dll] Setting dll directory to C:\Windows\SysWOW64\inetsrv
[2024-11-21T05:49:37.957Z, PID: 6288] [aspnetcorev2_inprocess.dll] Setting current directory to h:\root\home\henryxxx\www\homenew\
[2024-11-21T05:49:40.131Z, PID: 6288] [aspnetcorev2_inprocess.dll] In-process callbacks set
[2024-11-21T05:49:40.150Z, PID: 6288] [aspnetcorev2_inprocess.dll] Event Log: 'Application 'h:\root\home\henryxxx\www\homenew\' started successfully.' 
End Event Log Message.
[2024-11-21T05:49:40.165Z, PID: 6288] [aspnetcorev2_inprocess.dll] Adding request. Total Request Count 1
[2024-11-21T05:49:40.395Z, PID: 6288] [aspnetcorev2_inprocess.dll] Removing Request 0

After WordPress triggers an HTTP Error 500

OutOfProcess hosting model
The processes:

Handles      WS(K)   CPU(s)     Id UserName  ProcessName
-------      -----   ------     -- --------  -----------
    169      47860     0.72  52028 henryxxx  php-cgi
    672      41392     0.47  67660 henryxxx  w3wp

The aspnetcore-debug log:

[2024-11-21T05:08:12.300Z, PID: 67660] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/henryxxx-SITE4'
[2024-11-21T05:08:12.538Z, PID: 67660] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalApplicationStop
[2024-11-21T05:08:12.555Z, PID: 67660] [aspnetcorev2.dll] Shutdown starting in 1000 ms.
[2024-11-21T05:08:13.574Z, PID: 67660] [aspnetcorev2.dll] Shutdown starting.
[2024-11-21T05:08:13.589Z, PID: 67660] [aspnetcorev2.dll] Stopping application '/LM/W3SVC/1074629900/ROOT'
[2024-11-21T05:08:13.603Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Stopping file watching.
[2024-11-21T05:08:13.623Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Waiting for file watcher thread to exit.
[2024-11-21T05:08:13.623Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Stopping file watcher thread
[2024-11-21T05:08:13.654Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Waiting for file watcher thread to exit.
[2024-11-21T05:08:13.676Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Event Log: 'Sent shutdown HTTP message to process with Id '32100' and received http status '202'.' 
End Event Log Message.
[2024-11-21T05:08:13.722Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Event Log: 'Application '/LM/W3SVC/1074629900/ROOT' with physical root 'h:\root\home\henryxxx\www\homenew\' shut down process with Id '32100' listening on port '26945'' 
End Event Log Message.
[2024-11-21T05:08:13.744Z, PID: 67660] [aspnetcorev2_outofprocess.dll] Canceling standard stream pipe reader.
[2024-11-21T05:08:20.297Z, PID: 67660] [aspnetcorev2.dll] Received a request during shutdown. Will return a 503 response.
[2024-11-21T05:08:20.314Z, PID: 67660] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:97 
[2024-11-21T05:08:20.333Z, PID: 67660] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:121 
[2024-11-21T05:08:20.447Z, PID: 67660] [aspnetcorev2.dll] Received a request during shutdown. Will return a 503 response.
[2024-11-21T05:08:20.494Z, PID: 67660] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:97 
[2024-11-21T05:08:20.510Z, PID: 67660] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:121

InProcess hosting model

Handles      WS(K)   CPU(s)     Id UserName  ProcessName
-------      -----   ------     -- --------  -----------
    142      13804     0.06  77200 henryxxx  conhost
    168      39240     0.22  10532 henryxxx  php-cgi
    168      48856     0.77  46932 henryxxx  php-cgi
    973     103784     2.70   6288 henryxxx  w3wp

[2024-11-21T05:51:16.078Z, PID: 6288] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/HENRYSTAFF-001-SITE4'
[2024-11-21T05:51:16.299Z, PID: 6288] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalApplicationStop
[2024-11-21T05:51:16.309Z, PID: 6288] [aspnetcorev2.dll] Shutdown starting in 1000 ms.
[2024-11-21T05:51:17.330Z, PID: 6288] [aspnetcorev2.dll] Shutdown starting.
[2024-11-21T05:51:17.340Z, PID: 6288] [aspnetcorev2.dll] Stopping application '/LM/W3SVC/1074629900/ROOT'
[2024-11-21T05:51:17.348Z, PID: 6288] [aspnetcorev2_inprocess.dll] Stopping file watching.
[2024-11-21T05:51:17.363Z, PID: 6288] [aspnetcorev2_inprocess.dll] Waiting for file watcher thread to exit.
[2024-11-21T05:51:17.363Z, PID: 6288] [aspnetcorev2_inprocess.dll] Stopping file watcher thread
[2024-11-21T05:51:17.393Z, PID: 6288] [aspnetcorev2_inprocess.dll] Waiting for file watcher thread to exit.
[2024-11-21T05:51:17.409Z, PID: 6288] [aspnetcorev2_inprocess.dll] Stopping CLR
[2024-11-21T05:51:17.423Z, PID: 6288] [aspnetcorev2_inprocess.dll] Starting shutdown sequence 0
[2024-11-21T05:51:17.423Z, PID: 6288] [aspnetcorev2_inprocess.dll] Drained all requests, notifying managed.
[2024-11-21T05:51:17.425Z, PID: 6288] [aspnetcorev2_inprocess.dll] Waiting for 0 requests to drain
[2024-11-21T05:51:17.470Z, PID: 6288] [aspnetcorev2_inprocess.dll] Managed application exited with code 0
[2024-11-21T05:51:17.481Z, PID: 6288] [aspnetcorev2_inprocess.dll] Clr thread wait ended: clrThreadExited: 1
[2024-11-21T05:51:17.491Z, PID: 6288] [aspnetcorev2_inprocess.dll] Event Log: 'Application 'MACHINE/WEBROOT/APPHOST/HENRYSTAFF-001-SITE1' has shutdown.' 
End Event Log Message.
[2024-11-21T05:51:17.507Z, PID: 6288] [aspnetcorev2_inprocess.dll] Canceling standard stream pipe reader
[2024-11-21T05:51:17.521Z, PID: 6288] [aspnetcorev2_inprocess.dll] Stopping in-process worker thread
[2024-11-21T05:51:30.327Z, PID: 6288] [aspnetcorev2.dll] Received a request during shutdown. Will return a 503 response.
[2024-11-21T05:51:30.336Z, PID: 6288] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:97 
[2024-11-21T05:51:30.345Z, PID: 6288] [aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\proxymodule.cpp:121

Expected Behavior

ASP.NET Core API can start again when a new request reaches to the site.

.NET Version

Host:
  Version:      9.0.0
  Architecture: x64
  Commit:       9d5a6a9aa4
  RID:          win-x64

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not  @dnfclas

@BrennanConroy Could you please look into this and help me troubleshoot the issue?

@matbech
Copy link
Author

matbech commented Nov 21, 2024 •
edited
Loading

@ZyuanC

I have two websites, ASP.NET Core API and WordPress, in the same application pool.

I had the same setup (fcgi/php and ASP.NET Core app in the same app pool) but then created a new application pool for the ASP.NET Core app and so far I haven't seen the issue again.

@dotnet-policy-service dotnet-policy-service bot added Needs: Attention 👋 This issue needs the attention of a contributor, typically because the OP has provided an update. and removed Needs: Author Feedback The author of this issue needs to respond in order for us to continue investigating this issue. labels Nov 21, 2024
@ZyuanC
Copy link

ZyuanC commented Nov 21, 2024

@matbech
Thank you for your update. I am afraid that if your ASP.NET Core app causes a server error that triggers the ANCM to shut down the process, you may encounter the same issue even though the app is in a single application pool. I expect the app to shut down and start properly, and I hope the .NET team can help identify where the problem is.

@BrennanConroy
Copy link
Member

[2024-11-21T05:51:16.078Z, PID: 6288] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/HENRYSTAFF-001-SITE4'
[Removed intermediate logs]
[2024-11-21T05:51:17.491Z, PID: 6288] [aspnetcorev2_inprocess.dll] Event Log: 'Application 'MACHINE/WEBROOT/APPHOST/HENRYSTAFF-001-SITE1' has shutdown.'

The application name HENRYSTAFF-001-SITE4 vs. HENRYSTAFF-001-SITE1 is suspicious. Is SITE4 the php/wordpress app and SITE1 the ASP.NET Core app? Wonder if we aren't filtering change notifications properly.

@ZyuanC
Copy link

ZyuanC commented Nov 22, 2024

@BrennanConroy
Yes, HENRYSTAFF-001-SITE4 is the WordPress app and HENRYSTAFF-001-SITE1 is the ASP.NET Core app. If there are any other confusions, please let me know. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions feature-iis Includes: IIS, ANCM Needs: Attention 👋 This issue needs the attention of a contributor, typically because the OP has provided an update.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gfoidl @matbech @BrennanConroy @ZyuanC