Skip to content
This repository has been archived by the owner on Jan 23, 2023. It is now read-only.
/ corefx Public archive

Fix ClientWebSocket close handshake when using proxy #36928

Merged
merged 2 commits into from
Apr 17, 2019

Conversation

davidsh
Copy link
Contributor

@davidsh davidsh commented Apr 16, 2019

When the client websocket was going through a proxy, it would sometimes transition to the
'Aborted' state and not the 'Closed' state after a successful closing handshake.

I first opened this bug a year ago but finally was able to get back to it and root cause the
problem. The problem only occured with the managed websocket. The .NET Framework did not have
this problem. And some proxies didn't cause a problem with managed websocket (such as Fiddler).

The root cause is a misinterpretation of RFC 6455, Section 7.1.1. This describes the behavior
of how the websocket's TCP connection should be closed. The most graceful way is to wait for
the server to initiate the close of the socket. In cases where it is taking too long to wait
for the server to start the TCP close, then the client can start the TCP close of the socket.

But no matter how the socket is finally closed, the websocket state should transition to 'Closed'
as soon as a valid closing handshake was performed (i.e. close frames both sent and received).
And this occurs before any logic for closing the TCP connection.

The code in managed websocket has a timer to wait 1 second for the server to start a close. If
the timer finishes, then the managed websocket calls an Abort() method to close the socket. This
ends up transitioning the websocket to an 'Aborted' state which is incorrect. The state should
only be moved to the 'Aborted' state if a closing handshake was not completed.

I added a new test to support this change and moved the LoopbackProxyServer code to Common.

Fixes #28027

When the client websocket was going through a proxy, it would sometimes transition to the
'Aborted' state and not the 'Closed' state after a successful closing handshake.

I first opened this bug a year ago but finally was able to get back to it and root cause the
problem. The problem only occured with the managed websocket. The .NET Framework did not have
this problem. And some proxies didn't cause a problem with managed websocket (such as Fiddler).

The root cause is a misinterpretation of RFC 6455, Section 7.1.1. This describes the behavior
of how the websocket's TCP connection should be closed. The most graceful way is to wait for
the server to initiate the close of the socket. In cases where it is taking too long to wait
for the server to start the TCP close, then the client can start the TCP close of the socket.

But no matter how the socket is finally closed, the websocket state should transition to 'Closed'
as soon as a valid closing handshake was performed (i.e. close frames both sent and received).
And this occurs before any logic for closing the TCP connection.

The code in managed websocket has a timer to wait 1 second for the server to start a close. If
the timer finishes, then the managed websocket calls an Abort() method to close the socket. This
ends up transitioning the websocket to an 'Aborted' state which is incorrect. The state should
only be moved to the 'Aborted' state if a closing handshake was not completed.

I added a new test to support this change and moved the LoopbackProxyServer code to Common.

Fixes #28027
@davidsh davidsh added this to the 3.0 milestone Apr 16, 2019
@davidsh davidsh self-assigned this Apr 16, 2019
@davidsh davidsh requested a review from a team April 16, 2019 17:07
@dotnet dotnet deleted a comment from azure-pipelines bot Apr 16, 2019
@dotnet dotnet deleted a comment from azure-pipelines bot Apr 16, 2019
@dotnet dotnet deleted a comment from azure-pipelines bot Apr 16, 2019
Copy link
Member

@wfurt wfurt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@davidsh
Copy link
Contributor Author

davidsh commented Apr 16, 2019

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 4 pipeline(s).

@dotnet dotnet deleted a comment from azure-pipelines bot Apr 16, 2019
@dotnet dotnet deleted a comment from azure-pipelines bot Apr 16, 2019
@dotnet dotnet deleted a comment from azure-pipelines bot Apr 16, 2019
@davidsh
Copy link
Contributor Author

davidsh commented Apr 16, 2019

@davidsh
Copy link
Contributor Author

davidsh commented Apr 16, 2019

@davidsh davidsh merged commit 0c9684c into dotnet:master Apr 17, 2019
@davidsh davidsh deleted the websocket_withproxy branch April 17, 2019 00:15
Copy link
Member

@stephentoub stephentoub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@stephentoub
Copy link
Member

Innerloop failures not related to PR:

@davidsh, they are related. Those HttpListener failures are asserts due to this change. All PRs are failing as a result.

Assertion Failed
Unexpected state Closed.



   at System.Net.WebSockets.ManagedWebSocket.CloseAsyncPrivate(WebSocketCloseStatus closeStatus, String statusDescription, CancellationToken cancellationToken) in D:\a\1\s\src\Common\src\System\Net\WebSockets\ManagedWebSocket.cs:line 1069

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 581

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\ExecutionContext.cs:line 172

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 617

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 596

   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\TaskContinuation.cs:line 794

   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\Task.cs:line 3335

   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\Future.cs:line 418

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 727

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 272

   at System.Net.WebSockets.ManagedWebSocket.SendCloseFrameAsync(WebSocketCloseStatus closeStatus, String closeStatusDescription, CancellationToken cancellationToken) in D:\a\1\s\src\Common\src\System\Net\WebSockets\ManagedWebSocket.cs:line 1174

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 581

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\ExecutionContext.cs:line 172

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 617

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 596

   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\TaskContinuation.cs:line 794

   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\Task.cs:line 3335

   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\Future.cs:line 418

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 727

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 272

   at System.Net.WebSockets.ManagedWebSocket.WaitForServerToCloseConnectionAsync(CancellationToken cancellationToken) in D:\a\1\s\src\Common\src\System\Net\WebSockets\ManagedWebSocket.cs:line 871

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 581

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\ExecutionContext.cs:line 172

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 617

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 596

   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\TaskContinuation.cs:line 794

   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\Task.cs:line 3335

   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\Future.cs:line 418

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 727

   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncValueTaskMethodBuilder.cs:line 161

   at System.Net.Http.HttpConnection.RawConnectionStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in D:\a\1\s\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RawConnectionStream.cs:line 68

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\ExecutionContext.cs:line 172

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 617

   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\TaskContinuation.cs:line 794

   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\Task.cs:line 3335

   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\Tasks\Future.cs:line 418

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 727

   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncValueTaskMethodBuilder.cs:line 161

   at System.Net.Http.HttpConnection.ReadAsync(Memory`1 destination) in D:\a\1\s\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnection.cs:line 1396

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\ExecutionContext.cs:line 172

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Runtime\CompilerServices\AsyncMethodBuilder.cs:line 617

   at System.Threading.ThreadPoolGlobals.<>c.<.cctor>b__5_0(Object state) in F:\workspace\_work\1\s\src\System.Private.CoreLib\shared\System\Threading\ThreadPool.cs:line 46

   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow) in D:\a\1\s\src\System.Net.Sockets\src\System\Net\Sockets\Socket.Tasks.cs:line 1048

   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _) in D:\a\1\s\src\System.Net.Sockets\src\System\Net\Sockets\Socket.Tasks.cs:line 869

   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags) in D:\a\1\s\src\System.Net.Sockets\src\System\Net\Sockets\SocketAsyncEventArgs.cs:line 781

   at System.Net.Sockets.SocketAsyncEventArgs.<>c.<.cctor>b__177_0(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped) in D:\a\1\s\src\System.Net.Sockets\src\System\Net\Sockets\SocketAsyncEventArgs.Windows.cs:line 1254

   at System.Threading.ThreadPoolBoundHandleOverlapped.CompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped) in F:\workspace\_work\1\s\src\System.Private.CoreLib\src\System\Threading\ClrThreadPoolBoundHandleOverlapped.cs:line 51

   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pNativeOverlapped) in F:\workspace\_work\1\s\src\System.Private.CoreLib\src\System\Threading\Overlapped.cs:line 65

@davidsh
Copy link
Contributor Author

davidsh commented Apr 17, 2019

@davidsh, they are related. Those HttpListener failures are asserts due to this change. All PRs are failing as a result.

Sigh. Ok. I'll get that fix/disabled asap.

davidsh added a commit to davidsh/corefx that referenced this pull request Apr 17, 2019
An HttpListener websocket test was failing after the change from PR dotnet#36928. That PR made changes
to tighten up the transition to the Closed state after the closing handshake had completed.

That PR missed some additional changes needed especially in cases where a server (such as loopback)
sends the close frame almost concurrently with the client sending the close frame.

This PR fixes the close frame handshake logic and also makes some optimizations in cases where the
websocket has already transitioned to the Aborted state during the closing handshake. In that case,
the websocket should not wait for a close frame to be received but should simply proceed to closing
the connection.

Fixes #36963
davidsh added a commit that referenced this pull request Apr 18, 2019
An HttpListener websocket test was failing after the change from PR #36928. That PR made changes
to tighten up the transition to the Closed state after the closing handshake had completed.

That PR missed some additional changes needed especially in cases where a server (such as loopback)
sends the close frame almost concurrently with the client sending the close frame.

This PR fixes the close frame handshake logic and also makes some optimizations in cases where the
websocket has already transitioned to the Aborted state during the closing handshake. In that case,
the websocket should not wait for a close frame to be received but should simply proceed to closing
the connection.

Fixes #36963
picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022
…6928)

When the client websocket was going through a proxy, it would sometimes transition to the
'Aborted' state and not the 'Closed' state after a successful closing handshake.

I first opened this bug a year ago but finally was able to get back to it and root cause the
problem. The problem only occured with the managed websocket. The .NET Framework did not have
this problem. And some proxies didn't cause a problem with managed websocket (such as Fiddler).

The root cause is a misinterpretation of RFC 6455, Section 7.1.1. This describes the behavior
of how the websocket's TCP connection should be closed. The most graceful way is to wait for
the server to initiate the close of the socket. In cases where it is taking too long to wait
for the server to start the TCP close, then the client can start the TCP close of the socket.

But no matter how the socket is finally closed, the websocket state should transition to 'Closed'
as soon as a valid closing handshake was performed (i.e. close frames both sent and received).
And this occurs before any logic for closing the TCP connection.

The code in managed websocket has a timer to wait 1 second for the server to start a close. If
the timer finishes, then the managed websocket calls an Abort() method to close the socket. This
ends up transitioning the websocket to an 'Aborted' state which is incorrect. The state should
only be moved to the 'Aborted' state if a closing handshake was not completed.

I added a new test to support this change and moved the LoopbackProxyServer code to Common.

Fixes dotnet/corefx#28027


Commit migrated from dotnet/corefx@0c9684c
picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022
An HttpListener websocket test was failing after the change from PR dotnet/corefx#36928. That PR made changes
to tighten up the transition to the Closed state after the closing handshake had completed.

That PR missed some additional changes needed especially in cases where a server (such as loopback)
sends the close frame almost concurrently with the client sending the close frame.

This PR fixes the close frame handshake logic and also makes some optimizations in cases where the
websocket has already transitioned to the Aborted state during the closing handshake. In that case,
the websocket should not wait for a close frame to be received but should simply proceed to closing
the connection.

Fixes dotnet/corefx#36963

Commit migrated from dotnet/corefx@1824535
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ClientWebSocket aborts on closes when going thru a proxy
3 participants