runtime: callbackUpdateSystemStack throw with Windows callback after CL 584597

[johnb8]

Go version

go version go1.23.2 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/john/.cache/go-build'
GOENV='/home/john/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/john/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/john/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.2'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/john/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK='/home/john/dev/ps-core-golang/go.work'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2203588374=/tmp/go-build -gno-record-gcc-switches'

What did you do?

When running my program built with Go version 1.23.2 it crashes with fatal error: runtime: stack split at bad time. This doesn't happen when the program is built with Go 1.22.4. I'm cross compiling for Windows from Linux. I've gone back through the releases and found that Go 1.22.5 is the version it starts crashing with.

I've looked through the diff between 1.22.4 and 1.22.5 and suspect this is the commit that introduced it 3560cf0 but I'm not familiar enough with Go internals to make any suggestions.

I've also noticed that x_cgo_getstackbound is a noop on Windows, not sure if that has anything to do with it.

Stacktrace

runtime: newstack at runtime.printlock+0x55 sp=0xc0001ef580 stack=[0xc0001ec000, 0xc0001f0000]
        morebuf={pc:0xed8994 sp:0xc0001ef588 lr:0x0}
        sched={pc:0xf0ccd5 sp:0xc0001ef580 lr:0x0 ctxt:0x0}
runtime.callbackUpdateSystemStack(0xc00006a008, 0x6b1dff140, 0x0)
        /usr/local/go/src/runtime/cgocall.go:257 +0x94 fp=0xc0001ef5e8 sp=0xc0001ef588 pc=0xed8994
runtime.cgocallbackg(0xf287a0, 0x6b1dff180, 0x0)
        /usr/local/go/src/runtime/cgocall.go:322 +0x71 fp=0xc0001ef668 sp=0xc0001ef5e8 pc=0xed8bb1
runtime.cgocallbackg(0xf287a0, 0x6b1dff180, 0x0)
        <autogenerated>:1 +0x2e fp=0xc0001ef690 sp=0xc0001ef668 pc=0xf4c90e
runtime.cgocallback(0xc0001ef720, 0xf39632, 0xf49540)
        /usr/local/go/src/runtime/asm_amd64.s:1084 +0xec fp=0xc0001ef6b8 sp=0xc0001ef690 pc=0xf47b0c
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:479 +0x8 fp=0xc0001ef6c8 sp=0xc0001ef6b8 pc=0xf45d48
runtime.cgocall(0xf49540, 0xc00006a330)
        /usr/local/go/src/runtime/cgocall.go:185 +0x72 fp=0xc0001ef730 sp=0xc0001ef6c8 pc=0xf39632
syscall.SyscallN(0xc0001ef738?, {0xc0001907c0?, 0x142568f?, 0x17af020?})
        /usr/local/go/src/runtime/syscall_windows.go:519 +0x46 fp=0xc0001ef750 sp=0xc0001ef730 pc=0xf43a86
github.com/saltosystems/winrt-go/windows/foundation.(*IAsyncOperation).SetCompleted(0xc0001907a0?, 0xc0000089f0?)
        /go/pkg/mod/github.com/saltosystems/[email protected]/windows/foundation/iasyncoperation.go:35 +0x5d fp=0xc0001ef7a0 sp=0xc0001ef750 pc=0x142637d
tinygo.org/x/bluetooth.awaitAsyncOperation(0x1fcdbb00030, {0x19f7f07, 0x6c})
        /go/pkg/mod/github.com/johnb8/[email protected]/adapter_windows.go:55 +0x158 fp=0xc0001ef850 sp=0xc0001ef7a0 pc=0x1434fd8
tinygo.org/x/bluetooth.(*Adapter).AddService.func2(0x1424c80?, 0x1fcdbafc7d0, 0xc0001ef958?)
        /go/pkg/mod/github.com/johnb8/[email protected]/gatts_windows.go:129 +0x58 fp=0xc0001ef958 sp=0xc0001ef850 pc=0x1438418
github.com/saltosystems/winrt-go/windows/foundation.(*TypedEventHandler).Invoke(0x1fcdbae4f60, 0xf4366f?, 0x1fcdbafc7d0, 0x1fcdbb00980, 0xee50a5?, 0xc0001efab0?, 0x1fcdbae4f60?, 0x5?, 0xc0001efa28?, 0x1fcdbaff178, ...)
        /go/pkg/mod/github.com/saltosystems/[email protected]/windows/foundation/typedeventhandler.go:114 +0x47 fp=0xc0001ef980 sp=0xc0001ef958 pc=0x1426967
github.com/saltosystems/winrt-go/internal/delegate.invoke(0x1fcdbae4f60, 0x1fcdbafc7d0, 0x1fcdbb00980, 0x0, 0x0, 0x7ffcd02be2df, 0x1fcdbae4f60, 0x1fcdbae4c30, 0x1fcdbaff178, 0x7ffcd022de80)
        /go/pkg/mod/github.com/saltosystems/[email protected]/internal/delegate/delegate.go:113 +0xb6 fp=0xc0001ef9e8 sp=0xc0001ef980 pc=0x1424ff6
runtime.call128(0x0, 0x1a2e458, 0xc0001efaa8, 0x8, 0x8, 0x50, 0xc0001efcd8)
        /usr/local/go/src/runtime/asm_amd64.s:778 +0x5d fp=0xc0001efa78 sp=0xc0001ef9e8 pc=0xf4645d
runtime.callbackWrap(0x6b1dff740)
        /usr/local/go/src/runtime/syscall_windows.go:396 +0x205 fp=0xc0001efe50 sp=0xc0001efa78 pc=0xf289a5
runtime.cgocallbackg1(0xf287a0, 0x6b1dff740, 0x0)
        /usr/local/go/src/runtime/cgocall.go:442 +0x295 fp=0xc0001eff10 sp=0xc0001efe50 pc=0xed8fd5
runtime.cgocallbackg(0xf287a0, 0x6b1dff740, 0x0)
        /usr/local/go/src/runtime/cgocall.go:361 +0x145 fp=0xc0001eff90 sp=0xc0001eff10 pc=0xed8c85
runtime.cgocallbackg(0xf287a0, 0x6b1dff740, 0x0)
        <autogenerated>:1 +0x2e fp=0xc0001effb8 sp=0xc0001eff90 pc=0xf4c90e
runtime.cgocallback(0x0, 0x0, 0x0)
        /usr/local/go/src/runtime/asm_amd64.s:1084 +0xec fp=0xc0001effe0 sp=0xc0001effb8 pc=0xf47b0c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001effe8 sp=0xc0001effe0 pc=0xf47d81
fatal error: runtime: stack split at bad time

What did you see happen?

Program crashes with fatal error: runtime: stack split at bad time when built with Go 1.23.2 but not when built with 1.22.4

What did you expect to see?

Program does not crash when built with Go 1.23.2

[gabyhelp]

Related Issues and Documentation

• x/mobile: bind with go1.22.5,1.22.6 leads to: fatal error: runtime: stack split at bad time #68760
• runtime: `hello world` build failure on windows/amd64 with go1.21.0 #61980 (closed)
• runtime: mcall called on m->g0 stack #56774 (closed)
• runtime: problem with preemption on windows arm32 #68552
• runtime: stack split at bad time when disable inlining #20510 (closed)
• runtime: infinite recursion on windows triggered by morestack #21382 (closed)
• Compilation finished with exit code 2 for go 1.18.1 #53950 (closed)
• runtime: unexpected return pc for runtime.systemstack_switch #35592 (closed)
• cmd/link: panic opening c-shared DLL on windows #26084 (closed)
• x/sys/unix: crash during initialization #51943 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[randall77]

@prattmic Somehow related to CL 584597?

[prattmic]

Seems possible.

This program seems to use Windows API callbacks (https://github.com/saltosystems/winrt-go/blob/main/windows/foundation/asyncoperationcompletedhandler.go#L45), which should be OK, but perhaps something odd is going on.

@johnb8 Is is possible for you to try https://go.dev/cl/600296 (from #68285) to see if it resolves the issue?

cc @cherrymui

[johnb8]

I was able to try a build with https://go.dev/cl/600296 and it did resolve the issue. Thanks for investigating this @prattmic!

[cherrymui]

CL 600296 is submitted. I guess we can call this fixed. Feel free to reopen if there is any more issues. Thanks.