Error when iterating orcgen.ConvertWebpage (panic: navigation failed: net::ERR_CONNECTION_REFUSED) #8

magandrez · 2024-10-01T08:59:21Z

Hi again,

So, I have been back working with this library and after a small refactor where I process a file to generate a PDF, I loop over a list of files on a folder generating HTMLs and from there, the PDF versions of those.

For this, I use ConvertWebpage as it is now implemented with the new configuration handler. So what I do is the following:

func (c *Certificate) RenderCertAsPDF(srcURL string, dst string) error {

	f, err := orcgen.ConvertWebpage(pdf.New().SetConfig(proto.PagePrintToPDF{
		DisplayHeaderFooter: false,
		PreferCSSPageSize:   true,
	}), srcURL)
	if err != nil {
		return fmt.Errorf("converting HTML to PDF: %v", err)
	}
	return f.Output(dst)
}

This is call within a loop, where before each call to RenderCertAsPDF I serve the content I need to render as PDF like:

func Serve(content []byte, assets *embed.FS) {
	mux := http.NewServeMux()
	mux.Handle("/", contentHandler(content))
	fs := http.FS(assets)
	mux.Handle("/static/assets/", http.FileServer(fs))
	server.Handler = mux

	go func() {
		if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Fatalf("listen: %s\n", err)
		}
	}()
	log.Println("Server started.")
}

and immediately after the PDF is rendered, I shutdown the server:

// above this, call RenderCartAsPDF
	ctx := context.WithoutCancel(context.Background())
	if err := server.Shutdown(ctx); err != nil {
		log.Fatalf("Server Shutdown Failed:%+v", err)
	}

So, the interesting part is, the first file in the loop is processed correctly, an HTML is generated, and a PDF is rendered. As for the second, an HTML is generated, but PDF rendering fails. See below trace:

go clean
rm -rf output/*
rm -f cover.html cover.out
go build -o carbonaide-cli .
2024/10/01 11:06:55 Certificate generation started.
2024/10/01 11:06:55 HTML document generated successfully
2024/10/01 11:06:55 Server started.
2024/10/01 11:06:58 PDF document generated successfully.
2024/10/01 11:06:58 Server Stopped
2024/10/01 11:06:58 The generated certificate can be found at output/01-08-2024_13.02.37.csv/certificate.pdf
2024/10/01 11:06:58 HTML document generated successfully
2024/10/01 11:06:58 Server started.
panic: navigation failed: net::ERR_CONNECTION_REFUSED

goroutine 1 [running]:
github.com/go-rod/rod/lib/utils.init.func2({0xb11da0?, 0xc003780990?})
	/home/manuel/workbench/go/pkg/mod/github.com/go-rod/[email protected]/lib/utils/utils.go:68 +0x1d
github.com/luabagg/orcgen/v2/pkg/webdriver.(*WebDriver).Connect.New.(*Browser).WithPanic.genE.func1({0xc003780a50?, 0x15?, 0x0?})
	/home/manuel/workbench/go/pkg/mod/github.com/go-rod/[email protected]/must.go:36 +0x62
github.com/go-rod/rod.(*Browser).MustPage(0xc003558360, {0xc000037668?, 0x0?, 0xc003552570?})
	/home/manuel/workbench/go/pkg/mod/github.com/go-rod/[email protected]/must.go:71 +0xd2
github.com/luabagg/orcgen/v2/pkg/webdriver.(*WebDriver).UrlToPage(0xc0017a40a8, {0xbf2c3c, 0x15})
	/home/manuel/workbench/go/pkg/mod/github.com/luabagg/orcgen/[email protected]/pkg/webdriver/webdriver.go:61 +0x73
github.com/luabagg/orcgen/v2.ConvertWebpage[...]({0xf58358?, 0xc0036f1330}, {0xbf2c3c, 0x15})
	/home/manuel/workbench/go/pkg/mod/github.com/luabagg/orcgen/[email protected]/orcgen.go:84 +0x89
github.com/carbonaide/carbonaide-cli/cmd/app/certificate.(*Certificate).RenderCertAsPDF(0xbe93f3?, {0xbf2c3c, 0x15}, {0xc0036b1380, 0x2e})
	/home/manuel/workbench/work/carbonaide-cli/cmd/app/certificate/certificate.go:77 +0x1d8
github.com/carbonaide/carbonaide-cli/cmd.generateCertificate({0xc00002c4e0, 0x1d})
	/home/manuel/workbench/work/carbonaide-cli/cmd/certify.go:153 +0xa1b
github.com/carbonaide/carbonaide-cli/cmd.init.func2(0x1897aa0, {0xbe6e92?, 0x4?, 0xbe6e96?})
	/home/manuel/workbench/work/carbonaide-cli/cmd/certify.go:74 +0x27b
github.com/spf13/cobra.(*Command).execute(0x1897aa0, {0xc00007a660, 0x2, 0x2})
	/home/manuel/workbench/go/pkg/mod/github.com/spf13/[email protected]/command.go:985 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0x1897d80)
	/home/manuel/workbench/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	/home/manuel/workbench/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
github.com/carbonaide/carbonaide-cli/cmd.Execute()
	/home/manuel/workbench/work/carbonaide-cli/cmd/root.go:55 +0x185
main.main()
	/home/manuel/workbench/work/carbonaide-cli/main.go:12 +0xf

I have been scratching my head around this and trying different approaches, but end on the same type of error. I did experiment with different ways of shutting down the server, but having come across the same problem, I am failing at debugging the source problem. I am unsure as to how to proceed debugging this issue. Could you give some insight on it? Is this a problem with orcgen not waiting enough time for the browser to request content to the server or is the server not behaving as expected? (I realized orcgen always initializes rod with the same idle and timeouts)

The text was updated successfully, but these errors were encountered:

luabagg · 2024-10-01T19:06:07Z

I'll take a look at this.

The webdriver can also be initialized from any given config if needed (FromConfig function), and you can also access the Browser instance, if you need specific configs:

But it seems that this issue occurs after the timeout. For example, I tried accessing x.com (that is now blocked here in Brazil) and it gives me a similar error.

magandrez · 2024-10-02T11:07:20Z

I'll take a look at this.

The webdriver can also be initialized from any given config if needed (FromConfig function), and you can also access the Browser instance, if you need specific configs:

Yeah, but the webdriver is not exposed by orcgen, I believe.

But it seems that this issue occurs after the timeout. For example, I tried accessing x.com (that is now blocked here in Brazil) and it gives me a similar error.

Please, let me know as soon as you know, I am working on offboarding myself from my current employer and I would like to close this topic and related development :)

luabagg · 2024-10-03T20:55:10Z

@magandrez it seems to be a problem with the gorod lib. I'll take a deeper look soon, try to update it, etc.

Thanks.

luabagg · 2024-10-13T15:36:43Z

@magandrez hey, sorry for the delay, I totally forgot your case.

Can you test it out? I downgraded go-rod to a version below and now it seems to be working fine. I'll also open an issue in go-rod.

luabagg · 2024-10-13T16:21:33Z

Here's the issue I submitted to go-rod - go-rod/rod#1123

I hope that the downgrade fix your issues.

magandrez · 2024-10-14T06:25:34Z

Here's the issue I submitted to go-rod - go-rod/rod#1123

I hope that the downgrade fix your issues.

Great!, I'll try with the downgraded version. Thanks for your help!

magandrez · 2024-10-14T07:36:16Z

It seems the problem is elsewhere. Utilizing the downgraded deps:

module github.com/carbonaide/carbonaide-cli

go 1.23

require (
	github.com/go-rod/rod v0.116.0
	github.com/luabagg/orcgen/v2 v2.0.2
	github.com/matryer/is v1.4.1
	github.com/spf13/cobra v1.8.1
	golang.org/x/exp v0.0.0-20241009180824-f66d83c29e7c
	gonum.org/v1/plot v0.14.0
	rsc.io/qr v0.2.0
)

require (
	git.sr.ht/~sbinet/gg v0.5.0 // indirect
	github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b // indirect
	github.com/campoy/embedmd v1.0.0 // indirect
	github.com/go-fonts/liberation v0.3.3 // indirect
	github.com/go-latex/latex v0.0.0-20240709081214-31cef3c7570e // indirect
	github.com/go-pdf/fpdf v0.9.0 // indirect
	github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/pmezard/go-difflib v1.0.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/ysmood/fetchup v0.2.4 // indirect
	github.com/ysmood/goob v0.4.0 // indirect
	github.com/ysmood/got v0.40.0 // indirect
	github.com/ysmood/gson v0.7.3 // indirect
	github.com/ysmood/leakless v0.9.0 // indirect
	golang.org/x/image v0.21.0 // indirect
	golang.org/x/net v0.30.0 // indirect
	golang.org/x/text v0.19.0 // indirect
)

The error remains the same:

2024/10/14 10:31:26 Certificate generation started.
2024/10/14 10:31:26 HTML document generated successfully
2024/10/14 10:31:26 Server started.
2024/10/14 10:31:29 PDF document generated successfully.
2024/10/14 10:31:29 Server Stopped
2024/10/14 10:31:29 The generated certificate can be found at output/01-08-2024_13.02.37.csv/certificate.pdf
2024/10/14 10:31:29 HTML document generated successfully
2024/10/14 10:31:29 Server started.
panic: navigation failed: net::ERR_CONNECTION_REFUSED

goroutine 1 [running]:
github.com/go-rod/rod/lib/utils.init.func2({0xb0f2e0?, 0xc000214400?})
	/home/manuel/workbench/go/pkg/mod/github.com/go-rod/[email protected]/lib/utils/utils.go:69 +0x1d
github.com/luabagg/orcgen/v2/pkg/webdriver.(*WebDriver).Connect.New.(*Browser).WithPanic.genE.func1({0xc0001861c0?, 0x15?, 0x0?})
	/home/manuel/workbench/go/pkg/mod/github.com/go-rod/[email protected]/must.go:36 +0x62
github.com/go-rod/rod.(*Browser).MustPage(0xc003462240, {0xc001085738?, 0xa26ffa?, 0xc000542450?})
	/home/manuel/workbench/go/pkg/mod/github.com/go-rod/[email protected]/must.go:71 +0xd2
github.com/luabagg/orcgen/v2/pkg/webdriver.(*WebDriver).UrlToPage(0xc005481650?, {0xbef854?, 0xc000500008?})
	/home/manuel/workbench/go/pkg/mod/github.com/luabagg/orcgen/[email protected]/pkg/webdriver/webdriver.go:60 +0x32
github.com/luabagg/orcgen/v2.ConvertWebpage[...]({0xf54c58?, 0xc00013ff10}, {0xbef854, 0x15})
	/home/manuel/workbench/go/pkg/mod/github.com/luabagg/orcgen/[email protected]/orcgen.go:88 +0x89
github.com/carbonaide/carbonaide-cli/cmd/app/certificate.(*Certificate).RenderCertAsPDF(0xbe6073?, {0xbef854, 0x15}, {0xc005481650, 0x2e})
	/home/manuel/workbench/work/carbonaide-cli/cmd/app/certificate/certificate.go:77 +0x112
github.com/carbonaide/carbonaide-cli/cmd.generateCertificate({0xc00002c4e0, 0x1d})
	/home/manuel/workbench/work/carbonaide-cli/cmd/certify.go:153 +0xa1b
github.com/carbonaide/carbonaide-cli/cmd.init.func2(0x1892a80, {0xbe3b12?, 0x4?, 0xbe3b16?})
	/home/manuel/workbench/work/carbonaide-cli/cmd/certify.go:74 +0x27b
github.com/spf13/cobra.(*Command).execute(0x1892a80, {0xc00007a660, 0x2, 0x2})
	/home/manuel/workbench/go/pkg/mod/github.com/spf13/[email protected]/command.go:985 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0x1892d60)
	/home/manuel/workbench/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	/home/manuel/workbench/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
github.com/carbonaide/carbonaide-cli/cmd.Execute()
	/home/manuel/workbench/work/carbonaide-cli/cmd/root.go:55 +0x185
main.main()
	/home/manuel/workbench/work/carbonaide-cli/main.go:12 +0xf

I will try to do some tests to find out more.

luabagg · 2024-10-14T11:28:09Z

Yeah, so with your problem, I ended up discovering another one, haha.

Maybe you can provide a sample code with a public URL for me because I couldn't simulate your error.

Or does it fail for any URLs? I didn't check it out.

magandrez · 2024-10-15T08:59:56Z

Yeah, so with your problem, I ended up discovering another one, haha.

Maybe you can provide a sample code with a public URL for me because I couldn't simulate your error.

Or does it fail for any URLs? I didn't check it out.

@luabagg

Here a sample code simplified, that makes go-rod fail: https://github.com/magandrez/go-rod-issue

luabagg · 2024-10-22T21:04:48Z

Hey, sorry for the delay, I've been really busy last few weeks

I'll try to fix this soon

luabagg · 2024-10-29T14:35:28Z

Bro, the problem is in your server implementation.

Checking the repo you opened, the server only works in the first time, then it stays loading. I didn't check a solution to update HTML, but I think that you may be able to do it without opening multiple servers.

Maybe you can do as following:
fs := http.FileServer(http.Dir("./static"))

Add all your HTML files to static, then iterate as following:

localhost:8080/file1.html
localhost:8080/file2.html
...

luabagg · 2024-11-19T23:04:18Z

Hey, any updates? I'm going to close this for now.

If there's any issue with the lib, feel free to open it again. Thanks.

luabagg closed this as completed Nov 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error when iterating orcgen.ConvertWebpage (panic: navigation failed: net::ERR_CONNECTION_REFUSED) #8

Error when iterating orcgen.ConvertWebpage (panic: navigation failed: net::ERR_CONNECTION_REFUSED) #8

magandrez commented Oct 1, 2024

luabagg commented Oct 1, 2024

magandrez commented Oct 2, 2024

luabagg commented Oct 3, 2024

luabagg commented Oct 13, 2024

luabagg commented Oct 13, 2024

magandrez commented Oct 14, 2024

magandrez commented Oct 14, 2024

luabagg commented Oct 14, 2024 •

edited

Loading

magandrez commented Oct 15, 2024

luabagg commented Oct 22, 2024

luabagg commented Oct 29, 2024

luabagg commented Nov 19, 2024

Error when iterating orcgen.ConvertWebpage (panic: navigation failed: net::ERR_CONNECTION_REFUSED) #8

Error when iterating orcgen.ConvertWebpage (panic: navigation failed: net::ERR_CONNECTION_REFUSED) #8

Comments

magandrez commented Oct 1, 2024

luabagg commented Oct 1, 2024

magandrez commented Oct 2, 2024

luabagg commented Oct 3, 2024

luabagg commented Oct 13, 2024

luabagg commented Oct 13, 2024

magandrez commented Oct 14, 2024

magandrez commented Oct 14, 2024

luabagg commented Oct 14, 2024 • edited Loading

magandrez commented Oct 15, 2024

luabagg commented Oct 22, 2024

luabagg commented Oct 29, 2024

luabagg commented Nov 19, 2024

luabagg commented Oct 14, 2024 •

edited

Loading