Varnish PHP Admin

A PHP-based Varnish Administration Tool

This tool will allow you to administer your varnish server via a web interface. It supports:

• Purging by URI.
• Banning with URI regex or full ban queries.
• Multiple/virtual host support for Top stats and actions.
• Optional connectivity over the Varnish administration port
• Display of top line stats from varnishstat with 3 different verbosity levels.
• Display of stat medians over 10, 30 and 60 minutes
• Display of top URIs, missed URIs and user agents.
• Password protection.

Installation

Requirements

Before installing, ensure you have the following:

• An Apache web server (at least version 2).
• PHP 5.6.
• sudo or root access to your webserver.
• An installation of Varnish (At least 4.0) with varnishstat and varnishtop.
• The ability to open ports, if you wish to use the admin connection type.
• An active, working Cron.
• Some knowledge of Varnish and VCL.

Before you begin

Varnish PHP Admin is built in such a way to to minimise its footprint and requirements set.

There are no external requirements beyond the minimum listed above. The PHP file is almost totally self-contained and uses the Bootstrap CDN for styling and interaction.

The exception being the settings.php file. This is a small file which returns an array for use by Varnish PHP Admin to define its environment variables. It is written in PHP to ensure that the settings cannot be accidentally viewed via the webserver, as well as allowing it to sit next to the main index.php file.

There is also a shell script which Varnish PHP Admin relies on to create the data with which it displays statistics. Without this, the PHP cannot directly access the data generated by varnishstat or varnishtop as it is rarely accessible by non eleveated users such as apache. It would also be unable to provide historical data.

1) Create the settings file.

For obvious reasons, the settings file does not come pre-populated. You must fill it in yourself.

Please place a file named settings.php next to the index.php file wherever it is installed. An example settings.php file is as follows:

<?php
return array(
	'password' => '',
	'varnish_data_path' => '/var/varnish/data',
	'varnish_socket_ip' => '127.0.0.1'
);

The following options are available:

Option	Default	Notes
password	''	Required: The password your Varnish PHP Admin will be protected with. This cannot be blank.
varnish_data_path	''	The root path defined by varnishstat.sh. More on this further down.
varnish_http_ip	::1	The IP address at which your Varnish server can be accessed. This is usually the same as your web server.
varnish_http_port	80	The port number at which your varnish server can be accessed. This is usually set to 80 to replace apache.
timezone	Europe/London	Set to make local to your area. (Full list of timezones).
varnish_ban_method	http	Defines the BAN-ing method. More on this below.
date_format	j F Y g:i a	Defines the format for outputted dates. More on date formatting.

Additional options for the admin connection type:

Option	Default	Notes
varnish_socket_ip	::1	Defines the IP address at which the Varnish admin can be found.
varnish_socket_port	6082	Defines the Varnish admin port.
varnish_socket_secret	''	Defines the contents of the Varnish secret file as set in your server's Varnish installation. More information on the secret file.

2) Decide on admin or http for the BAN-ing method

Varnish allows for an administration TCP port to be defined, through which you can connect via TELNET or similar and issue commands identical to those found the varnishadm CLI. This port allows for more effective banning solutions without VCL alterations as well as retrieval of a ban list, giving you an idea of the currently active bans.

If you want to use this method, the varnish daemon will need to be configured to open that port with the -T switch. More information on varnish daemon switches.

If however, you aren't able to open arbitrary ports or otherwise change the way varnish is loaded as a system executable, then use the http method, which will require alterations to the VCL in order to support banning.

Note: Even when using the admin connection type, PURGE requests will still be made over HTTP and the stats will only be available from the files collected with varnishstat.sh (See more on this below).

3) Set up varnishstat.sh and any virtual hosts

This is a simple shell script which provides two purposes:

1. To run as a root user, generating file-based data from varnishstat and varnishtop.
2. To provide historical data on a rolling 1 minute basis.

The idea behind this shell script is to run as the root user (or any user who can access the varnish administrative binaries as well as set permissions on the resulting files). It should run every 1 (one) minute from a crontab using a command similar to the following:

# assuming the script is placed in /var/varnish
  */1 *  *   *   *     /var/varnish/varnishstat.sh /var/varnish/data

The argument after the script filename defines where the data will be placed. The structure of the data is as follows:

/var/varnish/data/
 > top.txt
 > top-misses.txt
 > top-ua.txt
 > stat/
	 > ##.json

Each of the files generated within stat/ are based on the minute of the hour in which the script runs. They will roll over every 60 minutes. The Varnish PHP Admin script will use these stats to produce historical median data.

Virtual hosts

If your server has virtual hosts set up or is otherwise operating shared hosting, you can define a hosts path within the path you have defined to contain data. For example, to define two virtual hosts for the server, the following path structure should be used:

/var/varnish/data/
 > hosts/
   > mydomain.com
   > myotherdomain.com
   > subdomain.myotherdomain.com

If the hosts directory is found, the subfolders will be used to identify hostnames and data matching those hostnames for the "Top" statistics will be placed there.

The interface will also show a host switcher which can be used to filter the statistics and choose where the actions such as PURGE and BAN are sent to.

For every host you wish to show within the admin, the directories placed within hosts should match exactly by name the actual hostnames your server accepts.

4) Optionally set up the varnish VCL

If you've elected to use the http style method for BAN requests, or want to support PURGE requests in either case, then the VCL file controlling the Varnish installation will need configuration.

If you already have Varnish set up, you may already have a VCL file. If so, great! Read on. Otherwise, you should look into how VCL files work first.

Before supporting PURGE and/or BAN requests, an ACL should be created. You can do this with the following code:

# Before the subroutines
acl purge {
	# ACL we'll use later to allow PURGE or BAN
	"localhost";
	"127.0.0.1";
	"your.servers.external.ip";
	"::1";
}

You'll need to support PURGE requests in the VCL regardless of how you connect to varnish. This can be done with the following logic (although feel free to use your own):

sub vcl_recv {
	if (req.method == "PURGE") {
		if (!client.ip ~ purge) { # purge is the ACL defined at the begining
			# Not from an allowed IP? Then die with an error.
			return (synth(405, "This IP is not allowed to send PURGE requests."));
		}

		# If you got this stage (and didn't error out above), purge the cached result
		return (purge);
	}
}

In addition to that, you can add support for the BAN requests with some custom logic:

sub vcl_recv {
	if (req.method == "BAN") {
		if (!client.ip ~ purge) { # purge is the ACL defined at the begining
			# Not from an allowed IP? Then die with an error.
			return (synth(405, "This IP is not allowed to send BAN requests."));
		}

		if (req.http.Ban-Query-Full) {
			# A "full" ban query (the result of checking "Full Ban Query")
			ban(req.http.Ban-Query-Full);
			return (synth(200, "Full ban added"));
		} else if (req.http.Ban-Query) {
			# A normal URI based ban query
			ban("req.http.host == " + req.http.host + " && req.url ~ " + req.http.Ban-Query);
			return (synth(200, "Ban added"));
		} else {
			return (synth(405, "Ban query sent without Ban-Query or Ban-Query-Full headers."));
		}
	}
}

The above two snippets define two vcl_recv subroutines - This is intentional and done so for clarity within the documentation, although in reality while it is not essential, you may want to combine the above logic to support PURGE and BAN requests in one go.

Altering your .vcl file is completely optional, but without it you might not be able to use the PURGE/BAN features of the actions form.

Important note about HTTP based BANS in the Varnish PHP Admin

You have probably noticed above that in the example handling snippet there are references to headers named Ban-Query and Ban-Query-Full. These headers are sent by the Varnish PHP Admin whenever a BAN request is being performed. Due to the fact that the query could be any number of complex characters forming a posix-style regular expression or VCL logic, they are sent as URL-encoded headers in order to ensure they are transported to the Varnish interface correctly. You must add support for these headers if you want to use the HTTP based BAN feature of the actions form on the Varnish PHP Admin page.

4) Finish up

Once you have the index.php and settings.php in a publicly accessible location on your web server, and the varnishstat.sh file running on a 1 minute interval, everything should work as expected.

Using the Varnish PHP Admin

When you first load the admin page, you'll be prompted to enter your password. This will only happen once per session. Once you're in, you should see a screen similar to the following:

The interface is comprised of three main parts:

1. The actions form, which allows you to perform PURGE/BAN requests on the Varnish enabled server.
2. The Global Stats display, giving you a current (within ~1 minute) display of data as well as up to 60 minute median data.
3. The Ban list, which gives historical statistics on the top visited URIs, missed URIs and user agents. This will not show if using HTTP style banning.
4. The Top lists, which gives historical statistics on the top visited URIs, missed URIs and user agents.

Purging

Purging is easily done with the actions form:

1. Key in a URI into the Query field. This URI will be purged and the response will be shown below the form.
2. Press the "Purge" button to perform a PURGE operation on the active host.

Banning

While more powerful, banning is slightly more complicated.

Banning a URI

1. Enter a URI or posix-style regex to ban within the Query field. There are some examples below.
2. Press the "Ban" button to perform a BAN operation on the active host.

Banning based on a VCL query

This can be done by checking the Full ban query checkbox (and heeding the warning).

In this state, the Query field becomes the raw argument which will be sent to the ban function within the Varnish VCL subroutines. This allows you to perform queries with multiple arguments and logic.

If you are using virtual hosts or otherwise multiple hosts, you may want to define this within the ban query.

See Varnish-CLI for more information on ban queries.

Example ban URI regular expressions:

• Everything: .*
• Common inages: \.(gif|jpeg|jpg|png)(\?.*)?$
• CSS and LESS fies: \.(s?css|less)(\?.*)?$
• All JS files: \.(jsx?)(\?.*)?$

Displaying more information within Stats

If you want to see more information within the Stats panel, just choose one of the options from the 'Show' menu.

If a stat is not displayed even if 'Show' is set to 'All', then it is likely because the data is currently at zero, and it is being hidden.

Troubleshooting

Fatal Error! "The "settings.php" file does not exist. Have you created it?"

This one should be pretty clear already but if you're not sure, it's just a case of following step 1 of the installation to create the settings.php file which should be placed next to the index.php file wherever you've installed the Varnish PHP Admin page.

Fatal Error! "Socket could not be opened to host."

When PURGE-ing or BAN-ing over HTTP, Varnish PHP Admin will attempt to connect to your server using the IP and port you defined within settings.php. It also sends a Host header depdnding on the host you have selected to view.

If your varnish_socket_ip or varnish_socket_port options in the settings.php file are pointing to the wrong place, or the port defined is not opened by your firewall, the Varnish PHP Admin can have trouble creating a socket. Check that your server is accessible at the IP/port you have defined.

Fatal Error! "Socket error: [error message]"

If this occurs when using admin style BANs, you may have misconfigured the varnish_socket_ip or varnish_socket_port settings, or the IP and port combination could not be accessed via the Admin interface.

Check you have opened the defined port, and that the IP is accessible from within the server running the interface.

Fatal Error! "Password not defined. Please define a password before continuing!"

You must define a password option within the Varnish PHP Admin settings.php file. There are no restrictions on what this password can be, except that it cannot be empty.

Fatal Error! "Authentication failed." when banning

If you see this error and are using the admin ban method, you may have entered in an incorrect varnish_socket_secret option. The correct secret string can usually be found alongside your .vcl files in the /etc/varnish directory.

Stats or Top information are not displaying

The shell script varnishstat.sh attempts to ensure that the statistic files it creates are world-readable for you, but it likely needs to run as an elevated user in order to do so. This will also help ensure it can run the varnishstat and varnishlog commands as necessary.

Ensure you have the shell script set up as per step 3 of the installation instructions.

If everything is set up, you may also want to check the varnish_data_path option in the settings.php file is also pointing to the same path as the shell script command. For instance:

# in crontab:
/var/varnish/varnishstat.sh /some/path/to/my/data

# in settings.php:
	'varnish_data_path' => '/some/path/to/my/data',

The 'hosts' field in the header doesn't exist

If you are attempting to use virtual hosts then please ensure you have followed the guidance in step 3 regarding virtual hosts.

Directories must be placed within the hosts folder of your data path, and be readable by the user running the apache process. The shell script will take care of this but it must have run at least once.

PURGE requests don't work!

If your PURGE requests aren't giving a socket error but otherwise aren't working, check the following:

1. That you are selecting the correct host name to purge.
2. That you have entered the correct URI (without the hostname), starting with a slash, e.g: / would usually be the homepage.
3. That the .vcl file is set up to acccept PURGE requests and is accessible via the varnish_socket_ip and varnish_socket_port defined in your settings.php file.

BAN requests don't work!

If your BAN requests aren't giving a socket error but otherwise aren't working, check the following:

1. That you are selecting the correct host name to ban.
2. That you have entered the correct regular expression (without the hostname, starting with a slash, e.g: .*\.(jpg|jpeg) would ban all JPEG filenames.
3. That the .vcl file is set up to acccept BAN requests and is accessible via the varnish_socket_ip and varnish_socket_port defined in your settings.php file.
4. That your expression is not incorrectly formed in the case of sending a full ban request.

Any other errors, issues or questions

Feel free to start an issue and I'll get back to you within a reasonable time. Keep in mind that while I will likely be happy to help, this is an open source project and I have a day job.