Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extproc crashes on 403 forbidden error #419

Closed
yuzisun opened this issue Feb 25, 2025 · 2 comments · Fixed by #420
Closed

extproc crashes on 403 forbidden error #419

yuzisun opened this issue Feb 25, 2025 · 2 comments · Fixed by #420
Labels
bug Something isn't working

Comments

@yuzisun
Copy link
Contributor

yuzisun commented Feb 25, 2025

Description:
AI Gateway extproc crashes when getting 403 forbidden error, this is a regression from the recent PR f07a7ff

time=2025-02-23T18:27:15.814Z level=DEBUG msg="response headers processing" response_headers="headers:{key:\":status\" raw_value:\"403\"}"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xe81a2f]

goroutine 162 [running]:
github.com/envoyproxy/ai-gateway/internal/extproc.(*Server).processMsg(0xc0000c0048, {0x1357900, 0xc0003e6ae0}, {0x0, 0x0}, 0x15?)
	/Users/dsun20/go/src/github.com/envoyproxy/ai-gateway/internal/extproc/server.go:211 +0x44f
github.com/envoyproxy/ai-gateway/internal/extproc.(*Server).Process(0xc0000c0048, {0x135edf8, 0xc0000c61d0})
	/Users/dsun20/go/src/github.com/envoyproxy/ai-gateway/internal/extproc/server.go:169 +0x2a5
github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler({0x10e2280?, 0xc0000c0048}, {0x135be50, 0xc0000001c0})
	/Users/dsun20/go/pkg/mod/github.com/envoyproxy/go-control-plane/[email protected]/service/ext_proc/v3/external_processor_grpc.pb.go:106 +0xd8
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000396e00, {0x1357900, 0xc0003e6a20}, 0xc0002e49c0, 0xc0003b19e0, 0x1c6d380, 0x0)
	/Users/dsun20/go/pkg/mod/google.golang.org/[email protected]/server.go:1690 +0x1252
google.golang.org/grpc.(*Server).handleStream(0xc000396e00, {0x1357d08, 0xc000290820}, 0xc0002e49c0)
	/Users/dsun20/go/pkg/mod/google.golang.org/[email protected]/server.go:1814 +0xb47
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	/Users/dsun20/go/pkg/mod/google.golang.org/[email protected]/server.go:1030 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 161
	/Users/dsun20/go/pkg/mod/google.golang.org/[email protected]/server.go:1041 +0x11d

Repro steps:
The crash can be replicated if the ext authz filter returns 403, e.g user is not permissioned to access the model

Logs:

[2025-02-20T14:59:14.595Z] "POST /v1/chat/completions HTTP/1.1" 403 UAEX 129 0 7 "10.244.87.192" "curl/8.7.1" "8b08965a-0cde-4953-87f9-56918343b549" "llama-3-1-70b-instruct-vllm.example.com" "-" "envoy-gateway-eg-fef97bb3-76pvt"
@yuzisun yuzisun added the bug Something isn't working label Feb 25, 2025
@yuzisun
Copy link
Contributor Author

yuzisun commented Feb 25, 2025

cc @nacx @mathetake

@mathetake
Copy link
Member

lol i referenced the issue number 403 instead of 419

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants