Shale, a Clojure-backed Selenium hub replacement
Shale is a lightweight replacement for a Selenium hub. We've found hubs to be more trouble than they're worth operationally, but we still need some of the management features.
Currently shale lets you get, create, and delete webdriver sessions, and maintains "reservation" and "tag" metadata to make querying against sessions easy.
Nodes can be discovered via the AWS API, a list of URLs, or through a custom
INodePool
implementation. See the example config for details.
Clone the repo
git clone [email protected]:cardforcoin/shale.git
In development, you can run the service with lein
.
lein run
# OR, run it in a REPL
lein repl
> (init)
> (start)
INFO [shale.configurer] Loaded shale config...
{:node-list ["http://localhost:4444/wd/hub" "http://localhost:4443/wd/hub"],
:node-max-sessions 6,
:port 5000,
:webdriver-timeout 3000,
:start-webdriver-timeout 5000}
INFO [shale.nodes] - Starting the node pool...
INFO [shale.sessions] - Starting session pool...
INFO [shale.core] - Starting Jetty...
To deploy the service, make a jar and run that.
lein uberjar
java -jar target/shale-0.3.0-SNAPSHOT-standalone.jar
By default, a config file- config.clj
- is expected on the classpath.
example-config.clj
shows the various options. If you'd like to provide a full
path to an alternative config file, set an environment variable CONFIG_FILE
with the path, or include the option using another method supported by
environ.
All grid nodes should be configured so that they don't attempt to register with
a hub (register:false
in the node config), and should be accessible via the
machine running shale.
Session creation, selection, and modification uses an S-expression inspired syntax.
curl -d '{"require": ["browser_name", "phantomjs"]}' -XPOST \
http://localhost:5000/sessions/ \
--header "Content-Type:application/json"
{
"id": "05e9229d-356b-46a3-beae-f8ab02cea7db",
"webdriver_id": "bd416414-e80f-4556-9d97-ebcfdc44425f",
"reserved": false,
"node": {"url":"http://localhost:5555/wd/hub", "id":"2c28f0f2-e479-4501-a05d-a0991793abd7"},
"browser_name": "phantomjs",
"tags": []
}
curl http://localhost:5000/sessions/
[{
"id": "05e9229d-356b-46a3-beae-f8ab02cea7db",
"webdriver_id": "bd416414-e80f-4556-9d97-ebcfdc44425f",
"reserved": false,
"node": {"url":"http://localhost:5555/wd/hub", "id":"2c28f0f2-e479-4501-a05d-a0991793abd7"},
"browser_name": "phantomjs",
"tags": []
}]
curl -d '{"create": {"browser_name":"phantomjs", "tags":["walmart"]}}' \
-XPOST http://localhost:5000/sessions/ \
--header "Content-Type:application/json"
{
"id": "05e9229d-356b-46a3-beae-f8ab02cea7db",
"webdriver_id": "bd416414-e80f-4556-9d97-ebcfdc44425f",
"reserved": false,
"node": {"url":"http://localhost:5555/wd/hub", "id":"2c28f0f2-e479-4501-a05d-a0991793abd7"},
"browser_name": "phantomjs",
"tags": ["walmart"]
}
curl -d '{"require": ["and [["browser_name", "phantomjs"] ["tag", "walmart"]]],
"modify": [["reserve", true]]}' \
-XPOST http://localhost:5000/sessions/ \
--header "Content-Type:application/json"
{
"id": "05e9229d-356b-46a3-beae-f8ab02cea7db",
"webdriver_id": "bd416414-e80f-4556-9d97-ebcfdc44425f",
"reserved": true,
"node": {"url":"http://localhost:5555/wd/hub", "id":"2c28f0f2-e479-4501-a05d-a0991793abd7"},
"browser_name": "phantomjs",
"tags": ["walmart"]
}
curl -d '{"create": {"browser_name":"phantomjs", "node":{"id": "<node id>"}}}' \
-XPOST http://localhost:5000/sessions/ \
--header "Content-Type:application/json"
You can create sessions with custom options, like proxy settings, disabling
local storage, or any other settings exposed by Selenium's
DesiredCapabilities
.
curl -d '{"create": {"browser_name":"chrome", "extra_desired_capabilities": {"chromeOptions": {"args": ["--proxy-server=socks5://<host>:<port>", "--disable-plugins","--disable-local-storage"]}}}' \
-XPOST http://localhost:5000/sessions/ \
--header "Content-Type:application/json"
Note, though, that there are better ways to manage session proxies!
PATCH accepts an array of modifications, including change_tag
, reserve
, and
go_to_url
.
curl -d '[["change_tag", {"action": "add", "tag":"walmart"}], ["reserve", true]]' \
-XPATCH http://localhost:5000/sessions/05e9229d-356b-46a3-beae-f8ab02cea7db \
--header "Content-Type:application/json"
{
"id": "05e9229d-356b-46a3-beae-f8ab02cea7db",
"webdriver_id": "bd416414-e80f-4556-9d97-ebcfdc44425f",
"reserved": false,
"node": {"url":"http://localhost:5555/wd/hub", "id":"2c28f0f2-e479-4501-a05d-a0991793abd7"},
"browser_name": "phantomjs",
"tags": ["walmart", "logged-in"]
}
Note that this will de-allocate the Selenium driver.
curl -XDELETE http://localhost:5000/sessions/05e9229d-356b-46a3-beae-f8ab02cea7db
true
curl -XDELETE http://localhost:5000/sessions/
true
Shale also includes its own proxy management that allows proxies to be shared between sessions.
An initial proxy list can be provided in the config.
curl -XPOST http://localhost:5000/proxies/ \
-d '{
"public_ip": "8.8.8.8",
"host": "127.0.0.1",
"port": 1234,
"type":"socks5"
"shared":true
}' \
--header "Content-Type:application/json"
{
"id":"f7c64a2c-595d-434c-80f0-15c9751ddcc8",
"public_ip":"127.0.0.1",
"host":"128.0.0.1",
"port":1234,
"type":"socks5",
"active":true,
"shared":true
}
A proxy with "shared": false
can only be used by sessions that reference it
by ID. Those with "shared": true
are candidates for use by other sessions.
The public_ip
is an optional attribute to keep track of a proxies
public-facing IP. This can be useful if you're testing public sites on the
internet, and have an IP whitelist.
Currently, only SOCKS proxies are supported.
curl http://localhost:5000/proxies/
[
{
"id":"f7c64a2c-595d-434c-80f0-15c9751ddcc8",
"public_ip":"8.8.8.8",
"host":"127.0.0.1",
"port":1234,
"type":"socks5",
"active":true,
"shared":true
}
]
curl -d '{"create": {"browser_name":"chrome", "proxy": {"type": "socks5", "host": "<host>", "port":<port>, "shared": true}}}'
-XPOST http://localhost:5000/sessions/ \
--header "Content-Type:application/json"
If the proxy hasn't been seen before, it will be recorded as a candidate for use
by other sessions. To avoid that behavior, set "shared": false
.
curl -d '{"require": ["and" [["browser_name", "chrome"] ["proxy", ["id", "f7c64a2c-595d-434c-80f0-15c9751ddcc8"]]]]}'
-XPOST http://localhost:5000/sessions/ \
--header "Content-Type:application/json"
The Clojure client returns functional web drivers using clj-webdriver
,
and includes a macro to make working with drivers easier.
Here's an example of how to get-or-create, reserve, use, and release a driver
using the with-webdriver*
macro, inspired by the clj-webdriver
examples.
;; Log into Github
;;
(use '[shale.client :only [with-driver]])
(use 'clj-webdriver.taxi)
(with-webdriver* {:browser-name :firefox :tags ["github"]}
(to "https://github.com")
(click "a[href*='login']")
(input-text "#login_field" "your_username")
(-> "#password"
(input-text "your_password")
submit))
See the clj-webdriver docs and the client source for more details.
There is also a Python client with its own examples and documentation.