Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lefthook freezes under a large filesystem with constrained resources/security measures (GitLab CI/CD Shared Runners) #834

Open
sanmai-NL opened this issue Oct 7, 2024 · 6 comments
Open

Lefthook freezes under a large filesystem with constrained resources/security measures (GitLab CI/CD Shared Runners) #834

sanmai-NL opened this issue Oct 7, 2024 · 6 comments
Labels
bug Something isn't working

Comments

@sanmai-NL
Copy link
Contributor

sanmai-NL commented Oct 7, 2024 •
edited
Loading

🔧 Summary

I encounter 100 % reproducible freezes of our CI pipeline inside a ‘larger’ container image in a constrained CI environment. That are the discerning features, not OS or CPU ISA. The freeze doesn't occur on my laptop with the same image and filesystem.

Lefthook version

Version 1.7.16

Steps to reproduce

It can all be seen here https://gitlab.com/han-aim/template/base-images/-/jobs/8007847885

Expected results

Lefthook not freezing. It's not entirely clear to me whether all commands have been run before the freeze.

Actual results

A freeze. I generated a traceback after a delay to reach the frozen state:

Program received signal SIGQUIT, Quit.
0x0000fffff7e2cd8c in kill () from /lib/aarch64-linux-gnu/libc.so.6
SIGQUIT: quit
PC=0x82dbc m=0 sigcode=0
goroutine 0 gp=0x93dbc0 m=0 mp=0x93e440 [idle]:
runtime.futex(0x93e580, 0x80, 0x0, 0x0, 0x0, 0x0)
	runtime/sys_linux_arm64.s:651 +0x1c fp=0xffffffffc630 sp=0xffffffffc630 pc=0x82dbc
runtime.futexsleep(0xffffffffc6b8?, 0x5210c?, 0xffffffffc6b8?)
	runtime/os_linux.go:69 +0x2c fp=0xffffffffc680 sp=0xffffffffc630 pc=0x4749c
runtime.notesleep(0x93e580)
	runtime/lock_futex.go:170 +0x8c fp=0xffffffffc6c0 sp=0xffffffffc680 pc=0x1e72c
runtime.mPark(...)
	runtime/proc.go:1761
runtime.stopm()
	runtime/proc.go:[278](https://gitlab.com/han-aim/template/base-images/-/jobs/8007847885#L278)2 +0x84 fp=0xffffffffc6f0 sp=0xffffffffc6c0 pc=0x52124
runtime.findRunnable()
	runtime/proc.go:3512 +0xfe0 fp=0xffffffffc860 sp=0xffffffffc6f0 pc=0x540a0
runtime.schedule()
	runtime/proc.go:3868 +0x98 fp=0xffffffffc8a0 sp=0xffffffffc860 pc=0x554e8
runtime.park_m(0x4000002e00)
	runtime/proc.go:4036 +0x240 fp=0xffffffffc900 sp=0xffffffffc8a0 pc=0x55b80
runtime.mcall()
	runtime/asm_arm64.s:192 +0x54 fp=0xffffffffc910 sp=0xffffffffc900 pc=0x7f574
goroutine 1 gp=0x40000021c0 m=6 mp=0x4000342008 [syscall, 3 minutes]:
syscall.Syscall(0x3f, 0x3, 0x400037e044, 0x3bc)
	syscall/syscall_linux.go:69 +0x20 fp=0x4000278a30 sp=0x40002789d0 pc=0x98c30
syscall.read(0x400030ecc0?, {0x400037e044?, 0x4000278a01?, 0x11d978?})
	syscall/zsyscall_linux_arm64.go:736 +0x40 fp=0x4000278a70 sp=0x4000278a30 pc=0x96a90
syscall.Read(...)
	syscall/syscall_unix.go:181
internal/poll.ignoringEINTRIO(...)
	internal/poll/fd_unix.go:736
internal/poll.(*FD).Read(0x400030ecc0, {0x400037e044, 0x3bc, 0x3bc})
	internal/poll/fd_unix.go:160 +0x224 fp=0x4000278b10 sp=0x4000278a70 pc=0xbeea4
os.(*File).read(...)
	os/file_posix.go:29
os.(*File).Read(0x40002900d8, {0x400037e044?, 0x4000278b98?, 0x44?})
	os/file.go:118 +0x70 fp=0x4000278b50 sp=0x4000278b10 pc=0xc9a30
os.(*fileWithoutWriteTo).Read(0x4000300de0?, {0x400037e044?, 0x4000278bd8?, 0xb8408?})
	<autogenerated>:1 +0x2c fp=0x4000278b80 sp=0x4000278b50 pc=0xd22ac
bytes.(*Buffer).ReadFrom(0x4000300de0, {0x5d9f20, 0x40002900e8})
	bytes/buffer.go:211 +0x90 fp=0x4000278be0 sp=0x4000278b80 pc=0x11d760
io.copyBuffer({0x5da2e0, 0x4000300de0}, {0x5d9f20, 0x40002900e8}, {0x0, 0x0, 0x0})
	io/io.go:415 +0x14c fp=0x4000278c60 sp=0x4000278be0 pc=0xb828c
io.Copy(...)
	io/io.go:388
os.genericWriteTo(0x40002900d8?, {0x5da2e0, 0x4000300de0})
	os/file.go:269 +0x5c fp=0x4000278cb0 sp=0x4000278c60 pc=0xca3cc
os.(*File).WriteTo(0x40002900d8, {0x5da2e0, 0x4000300de0})
	os/file.go:247 +0xa0 fp=0x4000278cf0 sp=0x4000278cb0 pc=0xca2e0
io.copyBuffer({0x5da2e0, 0x4000300de0}, {0x5d9f80, 0x40002900d8}, {0x0, 0x0, 0x0})
	io/io.go:411 +0x98 fp=0x4000278d70 sp=0x4000278cf0 pc=0xb81d8
io.Copy(...)
	io/io.go:388
github.com/evilmartians/lefthook/internal/lefthook/runner/exec.CommandExecutor.execute({}, {0x5ddeb8?, 0x4000269200?}, {0x40001b1350?, 0x40003c3f80?}, 0x4000278f50)
	github.com/evilmartians/lefthook/internal/lefthook/runner/exec/execute_unix.go:98 +0x294 fp=0x4000278e50 sp=0x4000278d70 pc=0x40aec4
github.com/evilmartians/lefthook/internal/lefthook/runner/exec.CommandExecutor.Execute({}, {0x5ddeb8, 0x4000269200}, {{0x400045dec0, 0xe}, {0x4000309b90, 0x24}, {0x40003c3380, 0x1, 0x1}, ...}, ...)
	github.com/evilmartians/lefthook/internal/lefthook/runner/exec/execute_unix.go:69 +0x528 fp=0x4000[279](https://gitlab.com/han-aim/template/base-images/-/jobs/8007847885#L279)050 sp=0x4000278e50 pc=0x40aaa8
github.com/evilmartians/lefthook/internal/lefthook/runner/exec.(*CommandExecutor).Execute(0xe?, {0x5ddeb8?, 0x4000269200?}, {{0x400045dec0, 0xe}, {0x4000309b90, 0x24}, {0x40003c3380, 0x1, 0x1}, ...}, ...)
	<autogenerated>:1 +0xa8 fp=0x40002790e0 sp=0x4000279050 pc=0x40b0f8
github.com/evilmartians/lefthook/internal/lefthook/runner.(*Runner).run(0x40000a0410, {0x5ddeb8, 0x4000269200}, {{0x400045dec0, 0xe}, {0x4000309b90, 0x24}, {0x40003c3380, 0x1, 0x1}, ...}, ...)
	github.com/evilmartians/lefthook/internal/lefthook/runner/runner.go:538 +0x2a0 fp=0x40002791c0 sp=0x40002790e0 pc=0x411c20
github.com/evilmartians/lefthook/internal/lefthook/runner.(*Runner).runCommand(0x40000a0410, {0x5ddeb8, 0x4000269200}, {0x400045dec0, 0xe}, 0x40004ee680)
	github.com/evilmartians/lefthook/internal/lefthook/runner/runner.go:463 +0x1d0 fp=0x40002793e0 sp=0x40002791c0 pc=0x4113b0
github.com/evilmartians/lefthook/internal/lefthook/runner.(*Runner).runCommands(0x40000a0410, {0x5ddeb8, 0x4000269200})
	github.com/evilmartians/lefthook/internal/lefthook/runner/runner.go:421 +0x590 fp=0x40002796e0 sp=0x40002793e0 pc=0x410750
github.com/evilmartians/lefthook/internal/lefthook/runner.(*Runner).RunAll(0x40000a0410, {0x5ddeb8, 0x4000269200}, {0x4000279ac8, 0x2, 0x0?})
	github.com/evilmartians/lefthook/internal/lefthook/runner/runner.go:120 +0x41c fp=0x4000279800 sp=0x40002796e0 pc=0x40dc2c
github.com/evilmartians/lefthook/internal/lefthook.(*Lefthook).Run(0x40001b2450, {0xffffffffd06a, 0xc}, {0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0, ...}, ...}, ...)
	github.com/evilmartians/lefthook/internal/lefthook/run.go:173 +0xcd0 fp=0x4000279b50 sp=0x4000279800 pc=0x4164d0
github.com/evilmartians/lefthook/internal/lefthook.Run(0x0?, {0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, {0x0, ...}}, ...)
	github.com/evilmartians/lefthook/internal/lefthook/run.go:41 +0xb8 fp=0x4000279bd0 sp=0x4000279b50 pc=0x415778
github.com/evilmartians/lefthook/cmd.run.New.func3(0x40000da900?, {0x400007eee0?, 0x4?, 0x50022d?})
	github.com/evilmartians/lefthook/cmd/run.go:44 +0x94 fp=0x4000279c50 sp=0x4000279bd0 pc=0x4241f4
github.com/spf13/cobra.(*Command).execute(0x4000195208, {0x400007eec0, 0x2, 0x2})
	github.com/spf13/[email protected]/command.go:985 +0x840 fp=0x4000279de0 sp=0x4000279c50 pc=0x1cdb60
github.com/spf13/cobra.(*Command).ExecuteC(0x4000194308)
	github.com/spf13/[email protected]/command.go:1117 +0x344 fp=0x4000279ec0 sp=0x4000279de0 pc=0x1ce324
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:1041
github.com/evilmartians/lefthook/cmd.Lefthook()
	github.com/evilmartians/lefthook/cmd/lefthook.go:8 +0x20 fp=0x4000279f20 sp=0x4000279ec0 pc=0x423890
main.main()
	github.com/evilmartians/lefthook/main.go:10 +0x1c fp=0x4000279f40 sp=0x4000279f20 pc=0x42523c
runtime.main()
	runtime/proc.go:271 +0x28c fp=0x4000279fd0 sp=0x4000279f40 pc=0x4d0fc
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x4000279fd0 sp=0x4000279fd0 pc=0x81954
goroutine 2 gp=0x4000002700 m=nil [force gc (idle), 3 minutes]:
runtime.gopark(0x8a58d6ba8e?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x4000066f90 sp=0x4000066f70 pc=0x4d528
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.forcegchelper()
	runtime/proc.go:326 +0xb8 fp=0x4000066fd0 sp=0x4000066f90 pc=0x4d3b8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x4000066fd0 sp=0x4000066fd0 pc=0x81954
created by runtime.init.6 in goroutine 1
	runtime/proc.go:314 +0x24
goroutine 3 gp=0x4000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x4000067760 sp=0x4000067740 pc=0x4d528
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.bgsweep(0x400003a070)
	runtime/mgcsweep.go:318 +0x108 fp=0x40000677b0 sp=0x4000067760 pc=0x38c28
runtime.gcenable.gowrap1()
	runtime/mgc.go:203 +0x28 fp=0x40000677d0 sp=0x40000677b0 pc=0x2cdd8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40000677d0 sp=0x40000677d0 pc=0x81954
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:203 +0x6c
goroutine 4 gp=0x4000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0x3b9c5d21?, 0x3b9aca00?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x4000067f60 sp=0x4000067f40 pc=0x4d528
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.(*scavengerState).park(0x93d5e0)
	runtime/mgcscavenge.go:425 +0x5c fp=0x4000067f90 sp=0x4000067f60 pc=0x365bc
runtime.bgscavenge(0x400003a070)
	runtime/mgcscavenge.go:658 +0xac fp=0x4000067fb0 sp=0x4000067f90 pc=0x36b6c
runtime.gcenable.gowrap2()
	runtime/mgc.go:204 +0x28 fp=0x4000067fd0 sp=0x4000067fb0 pc=0x2cd78
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x4000067fd0 sp=0x4000067fd0 pc=0x81954
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:204 +0xac
goroutine 5 gp=0x4000003340 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x54b968?, 0x80?, 0x25?, 0x2000000020?)
	runtime/proc.go:402 +0xc8 fp=0x4000066580 sp=0x4000066560 pc=0x4d528
runtime.runfinq()
	runtime/mfinal.go:194 +0x108 fp=0x40000667d0 sp=0x4000066580 pc=0x2bea8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40000667d0 sp=0x40000667d0 pc=0x81954
created by runtime.createfing in goroutine 1
	runtime/mfinal.go:164 +0x80
goroutine 50 gp=0x4000003a40 m=nil [GC worker (idle), 3 minutes]:
runtime.gopark(0x6e3f4365cc?, 0x3?, 0x80?, 0x47?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x4000069730 sp=0x4000069710 pc=0x4d528
runtime.gcBgMarkWorker()
	runtime/mgc.go:1310 +0xd8 fp=0x40000697d0 sp=0x4000069730 pc=0x2eed8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40000697d0 sp=0x40000697d0 pc=0x81954
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1234 +0x28
goroutine 10 gp=0x40003341c0 m=nil [GC worker (idle), 3 minutes]:
runtime.gopark(0x8a58dbe266?, 0x3ecf00?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x40002d0730 sp=0x40002d0710 pc=0x4d528
runtime.gcBgMarkWorker()
	runtime/mgc.go:1310 +0xd8 fp=0x40002d07d0 sp=0x40002d0730 pc=0x2eed8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40002d07d0 sp=0x40002d07d0 pc=0x81954
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1234 +0x28
goroutine 11 gp=0x4000334700 m=nil [select, 3 minutes, locked to thread]:
runtime.gopark(0x4000064fa0?, 0x2?, 0x78?, 0x4e?, 0x4000064f8c?)
	runtime/proc.go:402 +0xc8 fp=0x4000064e30 sp=0x4000064e10 pc=0x4d528
runtime.selectgo(0x4000064fa0, 0x4000064f88, 0x0?, 0x0, 0x40002b42a0?, 0x1)
	runtime/select.go:327 +0x614 fp=0x4000064f40 sp=0x4000064e30 pc=0x5ff44
runtime.ensureSigM.func1()
	runtime/signal_unix.go:1034 +0x198 fp=0x4000064fd0 sp=0x4000064f40 pc=0x787d8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x4000064fd0 sp=0x4000064fd0 pc=0x81954
created by runtime.ensureSigM in goroutine 1
	runtime/signal_unix.go:1017 +0xd8
goroutine 9 gp=0x40001ae380 m=nil [GC worker (idle), 3 minutes]:
runtime.gopark(0x8a58dbe23e?, 0x3ecf00?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x40002d0f30 sp=0x40002d0f10 pc=0x4d528
runtime.gcBgMarkWorker()
	runtime/mgc.go:1310 +0xd8 fp=0x40002d0fd0 sp=0x40002d0f30 pc=0x2eed8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40002d0fd0 sp=0x40002d0fd0 pc=0x81954
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1234 +0x28
goroutine 49 gp=0x40001ae540 m=nil [GC worker (idle), 3 minutes]:
runtime.gopark(0x6e3f436964?, 0x3ecf00?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x4000068f30 sp=0x4000068f10 pc=0x4d528
runtime.gcBgMarkWorker()
	runtime/mgc.go:1310 +0xd8 fp=0x4000068fd0 sp=0x4000068f30 pc=0x2eed8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x4000068fd0 sp=0x4000068fd0 pc=0x81954
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1234 +0x28
goroutine 39 gp=0x400056a700 m=nil [GC worker (idle), 3 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x40002d1730 sp=0x40002d1710 pc=0x4d528
runtime.gcBgMarkWorker()
	runtime/mgc.go:1310 +0xd8 fp=0x40002d17d0 sp=0x40002d1730 pc=0x2eed8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40002d17d0 sp=0x40002d17d0 pc=0x81954
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1234 +0x28
goroutine 40 gp=0x400056a8c0 m=nil [GC worker (idle), 3 minutes]:
runtime.gopark(0x6e3f437a44?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x40002d1f30 sp=0x40002d1f10 pc=0x4d528
runtime.gcBgMarkWorker()
	runtime/mgc.go:1310 +0xd8 fp=0x40002d1fd0 sp=0x40002d1f30 pc=0x2eed8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40002d1fd0 sp=0x40002d1fd0 pc=0x81954
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1234 +0x28
goroutine 41 gp=0x400056aa80 m=nil [GC worker (idle), 3 minutes]:
runtime.gopark(0x6e3f437bfc?, 0x3?, 0xc8?, 0xa?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x40002d2730 sp=0x40002d2710 pc=0x4d528
runtime.gcBgMarkWorker()
	runtime/mgc.go:1310 +0xd8 fp=0x40002d27d0 sp=0x40002d2730 pc=0x2eed8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40002d27d0 sp=0x40002d27d0 pc=0x81954
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1234 +0x28
goroutine 42 gp=0x400056ac40 m=nil [GC worker (idle), 3 minutes]:
runtime.gopark(0x6e3f4263d4?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xc8 fp=0x40002d2f30 sp=0x40002d2f10 pc=0x4d528
runtime.gcBgMarkWorker()
	runtime/mgc.go:1310 +0xd8 fp=0x40002d2fd0 sp=0x40002d2f30 pc=0x2eed8
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40002d2fd0 sp=0x40002d2fd0 pc=0x81954
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1234 +0x28
goroutine 43 gp=0x400056b180 m=8 mp=0x40001ac808 [syscall, 3 minutes]:
runtime.notetsleepg(0x9aef80, 0xffffffffffffffff)
	runtime/lock_futex.go:246 +0x34 fp=0x40002d3f90 sp=0x40002d3f60 pc=0x1e9b4
os/signal.signal_recv()
	runtime/sigqueue.go:152 +0x30 fp=0x40002d3fb0 sp=0x40002d3f90 pc=0x7dfe0
os/signal.loop()
	os/signal/signal_unix.go:23 +0x1c fp=0x40002d3fd0 sp=0x40002d3fb0 pc=0x4081bc
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40002d3fd0 sp=0x40002d3fd0 pc=0x81954
created by os/signal.Notify.func1.1 in goroutine 1
	os/signal/signal.go:151 +0x28
goroutine 44 gp=0x400056b340 m=nil [select, 3 minutes]:
runtime.gopark(0x40000687a8?, 0x2?, 0x88?, 0x86?, 0x4000068794?)
	runtime/proc.go:402 +0xc8 fp=0x4000384e40 sp=0x4000384e20 pc=0x4d528
runtime.selectgo(0x4000384fa8, 0x4000068790, 0x4000088e40?, 0x0, 0x0?, 0x1)
	runtime/select.go:327 +0x614 fp=0x4000384f50 sp=0x4000384e40 pc=0x5ff44
os/signal.NotifyContext.func1()
	os/signal/signal.go:288 +0x68 fp=0x4000384fd0 sp=0x4000384f50 pc=0x407d78
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x4000384fd0 sp=0x4000384fd0 pc=0x81954
created by os/signal.NotifyContext in goroutine 1
	os/signal/signal.go:287 +0x148
goroutine 70 gp=0x4000334e00 m=nil [select, 3 minutes]:
runtime.gopark(0x40002ce730?, 0x2?, 0x0?, 0x0?, 0x40002ce6bc?)
	runtime/proc.go:402 +0xc8 fp=0x40002ce560 sp=0x40002ce540 pc=0x4d528
runtime.selectgo(0x40002ce730, 0x40002ce6b8, 0x0?, 0x1, 0x0?, 0x1)
	runtime/select.go:327 +0x614 fp=0x40002ce670 sp=0x40002ce560 pc=0x5ff44
os/exec.(*Cmd).watchCtx(0x40000cea80, 0x400034e180)
	os/exec/exec.go:773 +0x7c fp=0x40002ce7b0 sp=0x40002ce670 pc=0x3ed19c
os/exec.(*Cmd).Start.gowrap2()
	os/exec/exec.go:759 +0x2c fp=0x40002ce7d0 sp=0x40002ce7b0 pc=0x3ece9c
runtime.goexit({})
	runtime/asm_arm64.s:1222 +0x4 fp=0x40002ce7d0 sp=0x40002ce7d0 pc=0x81954
created by os/exec.(*Cmd).Start in goroutine 1
	os/exec/exec.go:759 +0x798
r0      0x93e580
r1      0x80
r2      0x0
r3      0x0
r4      0x0
r5      0x0
r6      0x4
r7      0x0
r8      0x62
r9      0x1
r10     0x0
r11     0x1
r12     0x28f
r13     0x7fffffff
r14     0x3429e
r15     0x3dfd24cc5
r16     0xfffffffeccf0
r17     0x27ffffcb
r18     0x0
r19     0x0
r20     0xffffffffc690
r21     0x93e440
r22     0x4000004000
r23     0x0
r24     0x8e41dc0be9fcd734
r25     0x40000472a8
r26     0x54bd98
r27     0x9[280](https://gitlab.com/han-aim/template/base-images/-/jobs/8007847885#L280)00
r28     0x93dbc0
r29     0xffffffffc628
lr      0x4749c
sp      0xffffffffc630
pc      0x82dbc
fault   0x0
#0  0x0000fffff7e2cd8c in kill () from /lib/aarch64-linux-gnu/libc.so.6
#1  0x0000aaaaaaaa52d0 in ?? ()
#2  <signal handler called>
#3  0x0000fffff7e2ce14 in sigsuspend () from /lib/aarch64-linux-gnu/libc.so.6
#4  0x0000aaaaaaaa2180 in ?? ()
#5  0x0000fffff7e184c4 in ?? () from /lib/aarch64-linux-gnu/libc.so.6
#6  0x0000fffff7e18598 in __libc_start_main () from /lib/aarch64-linux-gnu/libc.so.6
#7  0x0000aaaaaaaa[283](https://gitlab.com/han-aim/template/base-images/-/jobs/8007847885#L283)0 in ?? ()

Possible Solution

Either the filesystem is full or blocks an operation somehow (io.Copy activity) or some goroutine deadlock. I don't see why Lefthook itself would need to write to the filesystem.

Logs / Screenshots

LEFTHOOK_VERBOSE=true git ...
@sanmai-NL sanmai-NL added the bug Something isn't working label Oct 7, 2024
@sanmai-NL
Copy link
Contributor Author

I'm not sure this is related, but I closed stdin in the CI script to enforce non-interactivity, and it seems Go/how you use io.Copy does not safely handle that. golang/go#58628

@sanmai-NL
Copy link
Contributor Author

Can you consider setting https://pkg.go.dev/[email protected]#File.SetReadDeadline on p?

lefthook/internal/lefthook/runner/exec/execute_unix.go

Line 98 in 1c92f5b

_, _ = io.Copy(args.out, p)

@mrexox
Copy link
Member

mrexox commented Oct 7, 2024

Thank you for submitting this issue! When did you notice the freeze? Was it because of some lefthook configuration change, or CI runner changes?

I think I could try setting read timeout, not sure it will fix the issue (it will probably fail the command). From the traces I see that lefthook waits for the data from the pseudo terminal.

@sanmai-NL
Copy link
Contributor Author

sanmai-NL commented Oct 8, 2024 •
edited
Loading

There was no change. Immediately as I defined this CI job with this image, the failure started to happen consistently. Unlike with other images, which run the exact same CI job and Lefthook config.

@mrexox
Copy link
Member

mrexox commented Oct 9, 2024

I see, is there a way to reproduce this locally using Docker or something? Could you share the CI settings, like CPU and RAM available?

@sanmai-NL
Copy link
Contributor Author

sanmai-NL commented Oct 11, 2024 •
edited
Loading

@mrexox The CI Runner type (hardware) is based on a tag, it's on the right side of the CI job log I hyperlinked to. The images are public. However, the runners aren't that resource constrained I noticed. And it doesn't happen on my laptop. I think a security control or some unusual environmental factor, such as a closed terminal or so, causes this. Can you try to work out the pty robustness fixes in a PR? They would be good anyway, and maybe it's just that. I will test them for you if.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sanmai-NL @mrexox