Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Flaky Test]: TestUpgradeBrokenPackageVersion – unexpected fault address #5231

Open
rdner opened this issue Jul 31, 2024 · 8 comments
Open
Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team

Comments

@rdner
Copy link
Member

rdner commented Jul 31, 2024

Failing test case

TestUpgradeBrokenPackageVersion

Error message

unexpected fault address

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1663

OS

Windows

Stacktrace and notes

=== RUN   TestUpgradeBrokenPackageVersion
    upgrade_broken_package_test.go:59: Testing Elastic Agent upgrade from 8.16.0-SNAPSHOT to 8.14.3...
    fetcher.go:95: Using existing artifact elastic-agent-8.16.0-SNAPSHOT-windows-x86_64.zip
    fixture.go:282: Extracting artifact elastic-agent-8.16.0-SNAPSHOT-windows-x86_64.zip to C:\Users\windows\AppData\Local\Temp\TestUpgradeBrokenPackageVersion1935806727\001
    fixture.go:300: Completed extraction of artifact elastic-agent-8.16.0-SNAPSHOT-windows-x86_64.zip to C:\Users\windows\AppData\Local\Temp\TestUpgradeBrokenPackageVersion1935806727\001
    fixture.go:906: Components were not modified from the fetched artifact
    fetcher.go:95: Using existing artifact elastic-agent-8.14.3-windows-x86_64.zip
    fixture.go:282: Extracting artifact elastic-agent-8.14.3-windows-x86_64.zip to C:\Users\windows\AppData\Local\Temp\TestUpgradeBrokenPackageVersion1935806727\002
    fixture.go:300: Completed extraction of artifact elastic-agent-8.14.3-windows-x86_64.zip to C:\Users\windows\AppData\Local\Temp\TestUpgradeBrokenPackageVersion1935806727\002
    fixture.go:906: Components were not modified from the fetched artifact
    fixture.go:657: >> running binary with: [C:\Users\windows\AppData\Local\Temp\TestUpgradeBrokenPackageVersion1935806727\001\elastic-agent-8.16.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
    fixture.go:657: >> running binary with: [C:\Users\windows\AppData\Local\Temp\TestUpgradeBrokenPackageVersion1935806727\002\elastic-agent-8.14.3-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
    upgrader.go:217: installation of Elastic Agent will use --unprivileged as both start and end version support --unprivileged mode
    upgrader.go:252: Installing version "8.16.0-SNAPSHOT"
    fixture_install.go:164: [test TestUpgradeBrokenPackageVersion] Inside fixture install function
    fixture_install.go:192: [test TestUpgradeBrokenPackageVersion] Inside fixture installNoPkgManager function
    fixture.go:657: >> running binary with: [C:\Users\windows\AppData\Local\Temp\TestUpgradeBrokenPackageVersion1935806727\001\elastic-agent-8.16.0-SNAPSHOT-windows-x86_64\elastic-agent.exe install --force --non-interactive --unprivileged]
    fixture.go:657: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    upgrade_broken_package_test.go:83: package version files found: [data/elastic-agent-8.16.0-SNAPSHOT-2676a3/package.version]
    upgrade_broken_package_test.go:88: removing package version file "C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-8.16.0-SNAPSHOT-2676a3\\package.version"
    upgrader.go:296: Upgrading from version "8.16.0-SNAPSHOT-2676a3f372b3aeca96c266fcb18d633a6c416496" to version "8.14.3-2df2c10b10501e457b69ef94e0ada45304b2d1a9"
    fixture.go:657: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe upgrade 8.14.3 --source-uri file://C:\Temp\agent-upgrade-755662942 --skip-verify]
    upgrader.go:347: waiting for upgrade watcher to start
    upgrader.go:352: upgrade watcher started
    upgrader.go:357: Checking upgrade details state while Upgrade Watcher is running
    fixture.go:657: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe version --binary-only --yaml]
    fixture.go:657: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:657: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    upgrader.go:388: waiting for upgrade watcher to finish
    upgrader.go:393: upgrade watcher finished
    upgrader.go:398: Checking upgrade details state after successful upgrade
    fixture.go:657: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe version --binary-only --yaml]
    fixture.go:657: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:657: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture_install.go:265: [test TestUpgradeBrokenPackageVersion] Inside fixture cleanup function
    fixture.go:657: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe uninstall --force]
    fixture_install.go:316: 
        	Error Trace:	C:/Users/windows/agent/pkg/testing/fixture_install.go:316
        	            				C:/Program Files/Go/src/testing/testing.go:1175
        	            				C:/Program Files/Go/src/testing/testing.go:1353
        	            				C:/Program Files/Go/src/testing/testing.go:1683
        	Error:      	Received unexpected error:
        	            	error running uninstall command: exit status 2
        	Test:       	TestUpgradeBrokenPackageVersion
        	Messages:   	uninstalling agent failed. Output: "\r[    ] Stopping service  [0s] \r                              \r\r[ ===] Successfully stopped service  [0s] \r                                          \r\r[ ===] Stopping upgrade watcher; none found  [0s] \r                                                  \r\r[====] Removing service  [0s] \r                                                  \r\r[====] Successfully uninstalled service  [0s] \r                                                  \r\r[====] Removing install directory  [0s] unexpected fault address 0x27b4244\nfatal error: fault\n[signal 0xc0000005 code=0x1 addr=0x27b4244 pc=0xb0a362]\n\ngoroutine 24 gp=0xc0003e41c0 m=5 mp=0xc000100008 [running]:\nruntime.throw({0x27a9045?, 0x0?})\n\truntime/panic.go:1023 +0x65 fp=0xc0002a1878 sp=0xc0002a1848 pc=0xacf8a5\nruntime.sigpanic()\n\truntime/signal_windows.go:414 +0xd0 fp=0xc0002a18c0 sp=0xc0002a1878 pc=0xae69f0\nruntime.memmove()\n\truntime/memmove_amd64.s:169 +0x122 fp=0xc0002a18c8 sp=0xc0002a18c0 pc=0xb0a362\nbytes.(*Buffer).WriteString(0xe664c7?, {0x27ac0d3?, 0xaaa8e5?})\n\tbytes/buffer.go:193 +0xa8 fp=0xc0002a18f8 sp=0xc0002a18c8 pc=0xbc10c8\nnet/http.http2FrameHeader.writeDebug({0x76, 0x97, 0xb1, 0x0, 0x2adb740}, 0xc0005434f0)\n\tnet/http/h2_bundle.go:1588 +0xd8 fp=0xc0002a1960 sp=0xc0002a18f8 pc=0xe6f018\nruntime: g 24: unexpected return pc for net/http.http2FrameHeader.writeDebug called from 0xc0002a19f8\nstack: frame={sp:0xc0002a18f8, fp:0xc0002a1960} stack=[0xc0002a0000,0xc0002a2000)\n0x000000c0002a17f8:  0x000000c0002a1838  0x0000000000acfce5  \n0x000000c0002a1808:  0x000000c0002a1818  0x000000c0003e41c0 \n0x000000c0002a1818:  0x0000000000acfd20   0x000000c0003e41c0 \n0x000000c0002a1828:  0x0000000000acf8a5   0x000000c0002a1848 \n0x000000c0002a1838:  0x000000c0002a1868  0x0000000000acf8a5  \n0x000000c0002a1848:  0x000000c0002a1850  0x0000000000acf8c0  \n0x000000c0002a1858:  0x00000000027a9045  0x0000000000000005 \n0x000000c0002a1868:  0x000000c0002a18b0  0x0000000000ae69f0  \n0x000000c0002a1878:  0x00000000027a9045  0x0000000000000000 \n0x000000c0002a1888:  0x00000000027b4244  0x000000c0003e41c0 \n0x000000c0002a1898:  0x0000000066a67248  0x00000000079a38d0 \n0x000000c0002a18a8:  0x0000000000b76213   0x000000c0002a18e8 \n0x000000c0002a18b8:  0x0000000000b0a362   0x0000000000bc10c8  \n0x000000c0002a18c8:  0x0000000000e6496d   0xc1a1ba79879a38d0 \n0x000000c0002a18d8:  0x000000073a71159d  0x0000000000000007 \n0x000000c0002a18e8:  0x000000c0002a1ae0  0x0000000000e6f018  \n0x000000c0002a18f8: <0x0000000000e664c7   0x00000000027ac0d3 \n0x000000c0002a1908:  0x0000000000aaa8e5   0x0000000000000000 \n0x000000c0002a1918:  0x0000000000000000  0x0000000000000001 \n0x000000c0002a1928:  0x0000000000000010  0x0000000000000008 \n0x000000c0002a1938:  0x000000c0002a1990  0x0000000000aaabbd  \n0x000000c0002a1948:  0x000000c0002a19d0  0x0000000000aa24fb  \n0x000000c0002a1958:  0x000000c0002a19f8 >0x0000000000b19776  \n0x000000c0002a1968:  0x0000000002adb740  0x000000c0005434f0 \n0x000000c0002a1978:  0x0000000000000001  0x0000010000000008 \n0x000000c0002a1988:  0x0000023fee907698  0x000000073a71159d \n0x000000c0002a1998:  0xc1a1ba79879a38d0  0x0000000000000020 \n0x000000c0002a19a8:  0x000000c000100008  0x000000c00091c5a0 \n0x000000c0002a19b8:  0x0000000000000000  0x0000000000000000 \n0x000000c0002a19c8:  0x0000000000000000  0x0000000000000000 \n0x000000c0002a19d8:  0x0000000000000000  0x0000000000000000 \n0x000000c0002a19e8:  0x0000000000000000  0x000000c000100000 \n0x000000c0002a19f8:  0x000000c0002a1a40  0x0000000000b16325  \n0x000000c0002a1a08:  0x000000000000001a  0x0000000000000020 \n0x000000c0002a1a18:  0x000000000000000d  0x0000000000000020 \n0x000000c0002a1a28:  0x000000000000001a  0x0000000000000000 \n0x000000c0002a1a38:  0x000000c00091c5a0  0x000000c0002a1bc8 \n0x000000c0002a1a48:  0x0000000000c53974   0x000000c000975c60 \n0x000000c0002a1a58:  0x000000000000000d \ncreated by go.elastic.co/apm/transport.(*HTTPTransport).WatchConfig in goroutine 23\n\tgo.elastic.co/[email protected]/transport/http.go:390 +0xf1\n\ngoroutine 1 gp=0xc000078000 m=nil [sleep]:\nruntime.gopark(0x19a89a5dfdc?, 0x0?, 0xb8?, 0xf5?, 0x147cde5?)\n\truntime/proc.go:402 +0xce fp=0xc000a0f588 sp=0xc000a0f568 pc=0xad25ee\ntime.Sleep(0x1dcd6500)\n\truntime/time.go:195 +0x11a fp=0xc000a0f5c8 sp=0xc000a0f588 pc=0xb0631a\ngithub.com/elastic/elastic-agent/internal/pkg/agent/install.RemovePath({0xc0003302c0, 0x1e})\n\tgithub.com/elastic/elastic-agent/internal/pkg/agent/install/uninstall.go:186 +0x46 fp=0xc000a0f648 sp=0xc000a0f5c8 pc=0x147aba6\ngithub.com/elastic/elastic-agent/internal/pkg/agent/install.Uninstall({0xc00079a280, 0x30}, {0xc0003302c0, 0x1e}, {0x0, 0x0}, 0xc00033a4d0, 0xc00037e5a0)\n\tgithub.com/elastic/elastic-agent/internal/pkg/agent/install/uninstall.go:132 +0x65b fp=0xc000a0f958 sp=0xc000a0f648 pc=0x1479b9b\ngithub.com/elastic/elastic-agent/internal/pkg/agent/cmd.uninstallCmd(0xc000480660, 0xc0004b0f08)\n\tgithub.com/elastic/elastic-agent/internal/pkg/agent/cmd/uninstall.go:97 +0x545 fp=0xc000a0fbb8 sp=0xc000a0f958 pc=0x222f825\ngithub.com/elastic/elastic-agent/internal/pkg/agent/cmd.newUninstallCommandWithArgs.func1(0xc0004b2500?, {0x27a84da?, 0x4?, 0x27a83de?})\n\tgithub.com/elastic/elastic-agent/internal/pkg/agent/cmd/uninstall.go:30 +0x26 fp=0xc000a0fc38 sp=0xc000a0fbb8 pc=0x222f206\ngithub.com/spf13/cobra.(*Command).execute(0xc0004b0f08, {0xc00033a460, 0x1, 0x1})\n\tgithub.com/spf13/[email protected]/command.go:989 +0xab1 fp=0xc000a0fdc0 sp=0xc000a0fc38 pc=0xc913d1\ngithub.com/spf13/cobra.(*Command).ExecuteC(0xc0004b0008)\n\tgithub.com/spf13/[email protected]/command.go:1117 +0x3ff fp=0xc000a0fe98 sp=0xc000a0fdc0 pc=0xc91cbf\ngithub.com/spf13/cobra.(*Command).Execute(0xc000130040?)\n\tgithub.com/spf13/[email protected]/command.go:1041 +0x13 fp=0xc000a0feb0 sp=0xc000a0fe98 pc=0xc91813\nmain.main()\n\tgithub.com/elastic/elastic-agent/main.go:41 +0x213 fp=0xc000a0ff50 sp=0xc000a0feb0 pc=0x2234013\nruntime.main()\n\truntime/proc.go:271 +0x28b fp=0xc000a0ffe0 sp=0xc000a0ff50 pc=0xad21cb\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc000a0ffe8 sp=0xc000a0ffe0 pc=0xb09461\n\ngoroutine 2 gp=0xc000078700 m=nil [force gc (idle)]:\nruntime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)\n\truntime/proc.go:402 +0xce fp=0xc000081fa8 sp=0xc000081f88 pc=0xad25ee\nruntime.goparkunlock(...)\n\truntime/proc.go:408\nruntime.forcegchelper()\n\truntime/proc.go:326 +0xb8 fp=0xc000081fe0 sp=0xc000081fa8 pc=0xad2458\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc000081fe8 sp=0xc000081fe0 pc=0xb09461\ncreated by runtime.init.6 in goroutine 1\n\truntime/proc.go:314 +0x1a\n\ngoroutine 3 gp=0xc000078a80 m=nil [GC sweep wait]:\nruntime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)\n\truntime/proc.go:402 +0xce fp=0xc000083f80 sp=0xc000083f60 pc=0xad25ee\nruntime.goparkunlock(...)\n\truntime/proc.go:408\nruntime.bgsweep(0xc00007e000)\n\truntime/mgcsweep.go:318 +0xdf fp=0xc000083fc8 sp=0xc000083f80 pc=0xaba91f\nruntime.gcenable.gowrap1()\n\truntime/mgc.go:203 +0x25 fp=0xc000083fe0 sp=0xc000083fc8 pc=0xaaf1c5\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc000083fe8 sp=0xc000083fe0 pc=0xb09461\ncreated by runtime.gcenable in goroutine 1\n\truntime/mgc.go:203 +0x66\n\ngoroutine 4 gp=0xc000078c40 m=nil [GC scavenge wait]:\nruntime.gopark(0x10000?, 0x2ad54d8?, 0x0?, 0x0?, 0x0?)\n\truntime/proc.go:402 +0xce fp=0xc00008df78 sp=0xc00008df58 pc=0xad25ee\nruntime.goparkunlock(...)\n\truntime/proc.go:408\nruntime.(*scavengerState).park(0x3ccb2c0)\n\truntime/mgcscavenge.go:425 +0x49 fp=0xc00008dfa8 sp=0xc00008df78 pc=0xab82a9\nruntime.bgscavenge(0xc00007e000)\n\truntime/mgcscavenge.go:658 +0x59 fp=0xc00008dfc8 sp=0xc00008dfa8 pc=0xab8859\nruntime.gcenable.gowrap2()\n\truntime/mgc.go:204 +0x25 fp=0xc00008dfe0 sp=0xc00008dfc8 pc=0xaaf165\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0xb09461\ncreated by runtime.gcenable in goroutine 1\n\truntime/mgc.go:204 +0xa5\n\ngoroutine 18 gp=0xc0001041c0 m=nil [finalizer wait]:\nruntime.gopark(0xc000085e60?, 0xab777c?, 0x20?, 0x65?, 0x550011?)\n\truntime/proc.go:402 +0xce fp=0xc000085e20 sp=0xc000085e00 pc=0xad25ee\nruntime.runfinq()\n\truntime/mfinal.go:194 +0x107 fp=0xc000085fe0 sp=0xc000085e20 pc=0xaae247\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0xb09461\ncreated by runtime.createfing in goroutine 1\n\truntime/mfinal.go:164 +0x3d\n\ngoroutine 19 gp=0xc000200540 m=nil [GC worker (idle)]:\nruntime.gopark(0x19a7ed1b540?, 0x0?, 0x0?, 0x0?, 0x0?)\n\truntime/proc.go:402 +0xce fp=0xc000089f50 sp=0xc000089f30 pc=0xad25ee\nruntime.gcBgMarkWorker()\n\truntime/mgc.go:1310 +0xe5 fp=0xc000089fe0 sp=0xc000089f50 pc=0xab1305\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc000089fe8 sp=0xc000089fe0 pc=0xb09461\ncreated by runtime.gcBgMarkStartWorkers in goroutine 1\n\truntime/mgc.go:1234 +0x1c\n\ngoroutine 20 gp=0xc000200700 m=nil [GC worker (idle)]:\nruntime.gopark(0x19a7ec9bd04?, 0x0?, 0x0?, 0x0?, 0x0?)\n\truntime/proc.go:402 +0xce fp=0xc00008bf50 sp=0xc00008bf30 pc=0xad25ee\nruntime.gcBgMarkWorker()\n\truntime/mgc.go:1310 +0xe5 fp=0xc00008bfe0 sp=0xc00008bf50 pc=0xab1305\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0xb09461\ncreated by runtime.gcBgMarkStartWorkers in goroutine 1\n\truntime/mgc.go:1234 +0x1c\n\ngoroutine 21 gp=0xc0002008c0 m=nil [GC worker (idle)]:\nruntime.gopark(0x19a7ed1b540?, 0x0?, 0x0?, 0x0?, 0x0?)\n\truntime/proc.go:402 +0xce fp=0xc0002a5f50 sp=0xc0002a5f30 pc=0xad25ee\nruntime.gcBgMarkWorker()\n\truntime/mgc.go:1310 +0xe5 fp=0xc0002a5fe0 sp=0xc0002a5f50 pc=0xab1305\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc0002a5fe8 sp=0xc0002a5fe0 pc=0xb09461\ncreated by runtime.gcBgMarkStartWorkers in goroutine 1\n\truntime/mgc.go:1234 +0x1c\n\ngoroutine 22 gp=0xc000200a80 m=nil [GC worker (idle)]:\nruntime.gopark(0x19a7eb36bbc?, 0x0?, 0x0?, 0x0?, 0x0?)\n\truntime/proc.go:402 +0xce fp=0xc0002a7f50 sp=0xc0002a7f30 pc=0xad25ee\nruntime.gcBgMarkWorker()\n\truntime/mgc.go:1310 +0xe5 fp=0xc0002a7fe0 sp=0xc0002a7f50 pc=0xab1305\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc0002a7fe8 sp=0xc0002a7fe0 pc=0xb09461\ncreated by runtime.gcBgMarkStartWorkers in goroutine 1\n\truntime/mgc.go:1234 +0x1c\n\ngoroutine 23 gp=0xc000200c40 m=nil [select]:\nruntime.gopark(0xc000063d00?, 0x10?, 0x30?, 0x0?, 0xc000063880?)\n\truntime/proc.go:402 +0xce fp=0xc00058d6e0 sp=0xc00058d6c0 pc=0xad25ee\nruntime.selectgo(0xc00058dd00, 0xc000063860, 0xc000094190?, 0x0, 0x23fee90bbb8?, 0x1)\n\truntime/select.go:327 +0x725 fp=0xc00058d800 sp=0xc00058d6e0 pc=0xae40c5\ngo.elastic.co/apm.(*Tracer).loop(0xc00012b680)\n\tgo.elastic.co/[email protected]/tracer.go:975 +0xb5c fp=0xc00058dfc8 sp=0xc00058d800 pc=0x13afc3c\ngo.elastic.co/apm.newTracer.gowrap1()\n\tgo.elastic.co/[email protected]/tracer.go:496 +0x25 fp=0xc00058dfe0 sp=0xc00058dfc8 pc=0x13aba05\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc00058dfe8 sp=0xc00058dfe0 pc=0xb09461\ncreated by go.elastic.co/apm.newTracer in goroutine 1\n\tgo.elastic.co/[email protected]/tracer.go:496 +0xcbb\n\ngoroutine 34 gp=0xc000584000 m=nil [chan receive]:\nruntime.gopark(0x0?, 0x0?, 0x7a?, 0xeb?, 0x10100000000?)\n\truntime/proc.go:402 +0xce fp=0xc00058be28 sp=0xc00058be08 pc=0xad25ee\nruntime.chanrecv(0xc000580000, 0xc00058bf18, 0x1)\n\truntime/chan.go:583 +0x3cd fp=0xc00058bea0 sp=0xc00058be28 pc=0xa9a9ed\nruntime.chanrecv2(0x3c78040?, 0x24e5500?)\n\truntime/chan.go:447 +0x12 fp=0xc00058bec8 sp=0xc00058bea0 pc=0xa9a612\ngo.elastic.co/apm.(*Tracer).loop.func2()\n\tgo.elastic.co/[email protected]/tracer.go:878 +0x205 fp=0xc00058bfe0 sp=0xc00058bec8 pc=0x13b28a5\nruntime.goexit({})\n\truntime/asm_amd64.s:1695 +0x1 fp=0xc00058bfe8 sp=0xc00058bfe0 pc=0xb09461\ncreated by go.elastic.co/apm.(*Tracer).loop in goroutine 23\n\tgo.elastic.co/[email protected]/tracer.go:875 +0x3ac\n"
    fixture.go:1034: Dumping running processes in C:\Users\windows\agent\build\diagnostics\TEST-TestUpgradeBrokenPackageVersion-windows-amd64-ProcessDump-cleanup.json
    fixture.go:1037: failed to dump process; failed to create output file C:\Users\windows\agent\build\diagnostics\TEST-TestUpgradeBrokenPackageVersion-windows-amd64-ProcessDump-cleanup.json root: open C:\Users\windows\agent\build\diagnostics\TEST-TestUpgradeBrokenPackageVersion-windows-amd64-ProcessDump-cleanup.json: The system cannot find the path specified.
--- FAIL: TestUpgradeBrokenPackageVersion (162.36s)
@rdner rdner added Team:Elastic-Agent Label for the Agent team flaky-test Unstable or unreliable test cases. labels Jul 31, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

@cmacknz
Copy link
Member

cmacknz commented Jul 31, 2024

Here's the stack trace in that last line parsed out into a readable format:

stacktrace.txt

@cmacknz
Copy link
Member

cmacknz commented Jul 31, 2024

Here's the most relevant section:

runtime/panic.go:1023 +0x65 fp=0xc0002a1878 sp=0xc0002a1848 pc=0xacf8a5
runtime.sigpanic()
    runtime/signal_windows.go:414 +0xd0 fp=0xc0002a18c0 sp=0xc0002a1878 pc=0xae69f0
runtime.memmove()
    runtime/memmove_amd64.s:169 +0x122 fp=0xc0002a18c8 sp=0xc0002a18c0 pc=0xb0a362
bytes.(*Buffer).WriteString(0xe664c7?, {0x27ac0d3?, 0xaaa8e5?})
    bytes/buffer.go:193 +0xa8 fp=0xc0002a18f8 sp=0xc0002a18c8 pc=0xbc10c8
net/http.http2FrameHeader.writeDebug({0x76, 0x97, 0xb1, 0x0, 0x2adb740}, 0xc0005434f0)
    net/http/h2_bundle.go:1588 +0xd8 fp=0xc0002a1960 sp=0xc0002a18f8 pc=0xe6f018

runtime: g 24: unexpected return pc for net/http.http2FrameHeader.writeDebug called from 0xc0002a19f8
stack: frame={sp:0xc0002a18f8, fp:0xc0002a1960} stack=[0xc0002a0000,0xc0002a2000)
0x000000c0002a17f8:  0x000000c0002a1838  0x0000000000acfce5  
0x000000c0002a1808:  0x000000c0002a1818  0x000000c0003e41c0 
0x000000c0002a1818:  0x0000000000acfd20   0x000000c0003e41c0 
0x000000c0002a1828:  0x0000000000acf8a5   0x000000c0002a1848 
0x000000c0002a1838:  0x000000c0002a1868  0x0000000000acf8a5  
0x000000c0002a1848:  0x000000c0002a1850  0x0000000000acf8c0  
0x000000c0002a1858:  0x00000000027a9045  0x0000000000000005 
0x000000c0002a1868:  0x000000c0002a18b0  0x0000000000ae69f0  
0x000000c0002a1878:  0x00000000027a9045  0x0000000000000000 
0x000000c0002a1888:  0x00000000027b4244  0x000000c0003e41c0 
0x000000c0002a1898:  0x0000000066a67248  0x00000000079a38d0 
0x000000c0002a18a8:  0x0000000000b76213   0x000000c0002a18e8 
0x000000c0002a18b8:  0x0000000000b0a362   0x0000000000bc10c8  
0x000000c0002a18c8:  0x0000000000e6496d   0xc1a1ba79879a38d0 
0x000000c0002a18d8:  0x000000073a71159d  0x0000000000000007 
0x000000c0002a18e8:  0x000000c0002a1ae0  0x0000000000e6f018  
0x000000c0002a18f8: <0x0000000000e664c7   0x00000000027ac0d3 
0x000000c0002a1908:  0x0000000000aaa8e5   0x0000000000000000 
0x000000c0002a1918:  0x0000000000000000  0x0000000000000001 
0x000000c0002a1928:  0x0000000000000010  0x0000000000000008 
0x000000c0002a1938:  0x000000c0002a1990  0x0000000000aaabbd  
0x000000c0002a1948:  0x000000c0002a19d0  0x0000000000aa24fb  
0x000000c0002a1958:  0x000000c0002a19f8 >0x0000000000b19776  
0x000000c0002a1968:  0x0000000002adb740  0x000000c0005434f0 
0x000000c0002a1978:  0x0000000000000001  0x0000010000000008 
0x000000c0002a1988:  0x0000023fee907698  0x000000073a71159d 
0x000000c0002a1998:  0xc1a1ba79879a38d0  0x0000000000000020 
0x000000c0002a19a8:  0x000000c000100008  0x000000c00091c5a0 
0x000000c0002a19b8:  0x0000000000000000  0x0000000000000000 
0x000000c0002a19c8:  0x0000000000000000  0x0000000000000000 
0x000000c0002a19d8:  0x0000000000000000  0x0000000000000000 
0x000000c0002a19e8:  0x0000000000000000  0x000000c000100000 
0x000000c0002a19f8:  0x000000c0002a1a40  0x0000000000b16325  
0x000000c0002a1a08:  0x000000000000001a  0x0000000000000020 
0x000000c0002a1a18:  0x000000000000000d  0x0000000000000020 
0x000000c0002a1a28:  0x000000000000001a  0x0000000000000000 
0x000000c0002a1a38:  0x000000c00091c5a0  0x000000c0002a1bc8 
0x000000c0002a1a48:  0x0000000000c53974   0x000000c000975c60 
0x000000c0002a1a58:  0x000000000000000d 

created by go.elastic.co/apm/transport.(*HTTPTransport).WatchConfig in goroutine 23
    go.elastic.co/[email protected]/transport/http.go:390 +0xf1

goroutine 23 gp=0xc000200c40 m=nil [select]:
runtime.gopark(0xc000063d00?, 0x10?, 0x30?, 0x0?, 0xc000063880?)
    runtime/proc.go:402 +0xce fp=0xc00058d6e0 sp=0xc00058d6c0 pc=0xad25ee
runtime.selectgo(0xc00058dd00, 0xc000063860, 0xc000094190?, 0x0, 0x23fee90bbb8?, 0x1)
    runtime/select.go:327 +0x725 fp=0xc00058d800 sp=0xc00058d6e0 pc=0xae40c5
go.elastic.co/apm.(*Tracer).loop(0xc00012b680)
    go.elastic.co/[email protected]/tracer.go:975 +0xb5c fp=0xc00058dfc8 sp=0xc00058d800 pc=0x13afc3c
go.elastic.co/apm.newTracer.gowrap1()
    go.elastic.co/[email protected]/tracer.go:496 +0x25 fp=0xc00058dfe0 sp=0xc00058dfc8 pc=0x13aba05
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc00058dfe8 sp=0xc00058dfe0 pc=0xb09461
created by go.elastic.co/apm.newTracer in goroutine 1
    go.elastic.co/[email protected]/tracer.go:496 +0xcbb

@cmacknz
Copy link
Member

cmacknz commented Jul 31, 2024

It seems like the goroutine triggering the unexpected return pc here originated from within go.elastic.co/apm while we were in the process of uninstalling Elastic Agent on Windows.

It's not immediately clear that it's doing anything wrong since the offending function is in the net/http package but I'm going to ping @elastic/apm-agent-go anyway on the chance they may have seen this before.

@kruskall
Copy link
Member

go.elastic.co/[email protected]

go agent v1 is ancient. Aren't we using v2 in elastic agent or am I missing something ? 🤔

@axw
Copy link
Member

axw commented Aug 1, 2024

I don't recall having seen that panic with the agent before. I'm also a bit confused by the above; I only see /v2 libraries in go.mod, and go mod why -m go.elastic.co/apm comes up with nothing.

@simitt
Copy link
Contributor

simitt commented Aug 1, 2024

Took a quick look, and it seems that the PR to upgrade apm agent to v2 was merged 3 weeks ago (#4939), so 8.16.0-SNAPSHOT should be using the v2, while older versions are on v1.

Could there be something weird going on with the Elastic Agent upgrading from 8.16.0-SNAPSHOT to 8.14.3 ? (Is that the intended order?)

upgrade_broken_package_test.go:59: Testing Elastic Agent upgrade from 8.16.0-SNAPSHOT to 8.14.3...

@cmacknz
Copy link
Member

cmacknz commented Aug 1, 2024

Yes, two things:

  • We test upgrading from the two previous minors (8.14.x, 8.13.x) and the previous major (7.17) to the latest snapshot in a branch since this is what external users will be doing on the next release.
  • For 8.16.0-SNAPSHOT when we test our main branch, main may already be the latest commit in the SNAPSHOT, so to avoid upgrading to the same commit we go backwards. In this case it is picking the most recent official release which is guaranteed to be a different commit.

So go agent v1 will eventually age out of our tests but that will happen as minor releases happen.

I don't think we need any deeper investigation unless this starts happening a lot, and if it is a rare problem related to go agent v1 it will age out eventually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

No branches or pull requests

6 participants