Official port of whalesong

[parthibd]

It's mentioned in your comments you will officially drop support for FirefoxDriver at some point of time and move to pypeteer (perhaps?)

Instead of using selenium at all why don't we port whalesong totally to puppeteer ? I know it will be quiet and effort . But think about it , the whole async nature of node.js would benefit whalesong extensively .
Even though pypetter looks promising but we never know when that project gets abandoned . Any suggestion ?

Note : I have gone through whalesong-js . It's quiet good , I have used it .

[alfred82santa]

Well, I have an issue about to split code in two repositories. One for python code (It means whalesong core) and one for js code (it means js scriptlet). Once we split code, it would be easy to maintain more than one driver. I'm not really interested on js driver, but I understand others could be. In this case, the most important is to share scriptlet code.

About to drop Firefox driver, there are other solutions. Regular browsers consume too much memory and cpu. So, puppeteer is not a solution, either. We only need a webview where inject our scriptlet and a way to send results. Puppeteer or Pyppeteer (even using firefox, https://aslushnikov.github.io/ispuppeteerfirefoxready/) are not a good solution.

[parthibd]

Either way we need browser or webview automation as you are saying. What do you have in mind? Any libraries?

[parthibd]

I understand abandoning FirefoxDriver but I don't believe that there is any browser automation libraries out there as established as Puppeteer or Selenium, who happen to have a lot of support. If Selenium and Puppeteer is out of the question, what else is there?

[parthibd]

I don't think dropping selenium support is such a very good idea because you'll have to depend on puppeteer which I have looked into doesn't have a lot of active develop. Even though browsers can be too heavy for whalesong, I think we have not other option because one way or another we are automating a browser. And depending upon pypeteer will be like putting all the eggs in the same basket.

[alfred82santa]

We don't need any automation library... We just need a webview with an access point. Actually, with a few lines of code we could build a webview with QtWebKit or WebkitGtk (I don't like those solutions, either)... We don't need all methods designed for automation. We just need a way to inject our scriptlet and a way to send commands and to retrieve results. Whalesong is not an automation tool, we use them because it was easier at the beginning.

However, any backend solution will work with drivers build with python or javascript... or go... or rust...

About pyppeteer, you are right, it looks like it is dying... But the code actually is very simple, we could fork it and maintain what we need, just a few methods.

If you want to investigate more: I like WPEWebkit (low resources and headless mode) and Servo (it's new, maybe it's good).

[parthibd]

So you basically want to implement a custom webview to satisfy the requirements of whalesong ? That sounds like a good idea . And your'e right we need to make a separate repo for the scriplet . That way we can have different drivers . But one thing is for sure we can't be using selenium for much long .As I have tested with multiple instances with all the various optimizations possible firefox/chrome is a RAM and CPU hog .

And we can maintain a separate repo for pypeteer by forking it . But it will spread out the development time that could have gone into making whalesong awesome :) I like qtwebkit as I have done some development using it before .

And one reason not to use pypeteer is because it uses a patch version of chrome . In my experience using a patched version instead of a generally available one is not a very great idea . The problem starts to occur when you move from one platform to another linux to windows .

But for the time being sticking with Selenium is the only way until we decide to come up with replacement . So I hope you'll give it a thought before dropping FirefoxDriver altogether .

[alfred82santa]

Be aware, patched version of chrome used by pyppeteer is the same one used by puppeteer, it is maintained by google, I don't think it will be abandoned.

[alfred82santa]

We will not remove Firefox backend until we get a better solution, don't worry.

[parthibd]

I might be wrong but the chrome bundled with puppeteer is not a patched version . Its a version that works in the most stable manner with the API . I guess pypetter does the same .

[parthibd]

I love working with whalesong . And its already in a great shape and form . I wish to ensure continuous development of it . I'll look into webkit and qtwebview and see if I can get a simple webview for whalesong to serve its purposes.

[parthibd]

I ran some trials with webviews . I tried qtwebview , javafx webview , and even python webview without making an api for whalesong . I came to the conclusion that whenever a webview is launched it takes more or less the same amount of ram and cpu as firefox or chromium . Sure there can be optimizations .

The only ones that seem viable are WPEWebkit , servo (didnt try) . Maybe I am wrong but if what we want to achieve is using a simple webview we have to customize the rendering engine for the most optimal webview.