This lets you create, modify and destroy ZeroTier networks and members through Terraform.
Despite the ability to change the API endpoint, current version doesn't work with custom controllers because of API differences, so do not be fooled
Since this isn't maintained by Hashicorp, you have to install it manually. There are two main ways:
Download and unzip the latest release.
Then, move the binary to your terraform plugins directory. The docs don't fully describe where this is.
- On Mac, it's
~/.terraform.d/plugins/darwin_amd64
- On Linux, it's
~/.terraform.d/plugins/linux_amd64
- On Windows, it's
$APPDATA\terraform.d\plugins\windows_amd64
Install Go v1.12+ on your machine, clone the source,
and let make install
do the rest.
brew install go # or upgrade
git clone https://github.com/cormacrelf/terraform-provider-zerotier
cd terraform-provider-zerotier
make install
# it may take a while to download `hashicorp/terraform`. be patient.
Install go 1.12+ from your favourite package manager or from source. Then:
git clone https://github.com/cormacrelf/terraform-provider-zerotier
cd terraform-provider-zerotier
make install
# it may take a while to download `hashicorp/terraform`. be patient.
In PowerShell, running as Administrator:
choco install golang
choco install zip
choco install git # for git-bash
choco install make
In a shell that has Make, like Git-Bash:
git clone https://github.com/cormacrelf/terraform-provider-zerotier
cd terraform-provider-zerotier
make install
# it may take a while to download `hashicorp/terraform`. be patient.
Before you can use a new provider, you must run terraform init
in your
project, where the root .tf
file is.
Use export ZEROTIER_API_KEY="..."
, or define it in a provider block:
provider "zerotier" {
api_key = "..."
## Optinal: Override for DIY controller
## Could be overriden by ZEROTIER_CONTROLLER_URL env var or this block
## Defaults to https://my.zerotier.com/api when not provided
# controller_url = "https://my.zerotier.com/api"
}
To achieve a similar configuration to the ZeroTier default, do this:
variable "zt_cidr" { default = "10.0.96.0/24" }
resource "zerotier_network" "your_network" {
name = "your_network_name"
# auto-assign v4 addresses to devices
assignment_pool {
cidr = "${var.zt_cidr}"
}
# route requests to the cidr block on each device through zerotier
route {
target = "${var.zt_cidr}"
}
}
If you don't specify either an assignment pool or a managed route, while it's perfectly valid, your network won't be very useful, so try to do both.
You can have more than one assignment pool, and more than one route. Multiple routes are useful for connecting two networks together, like so:
variable "zt_cidr" { default = "10.96.0.0/24" }
variable "other_network" { default = "10.41.0.0/24" }
locals {
# the first address is reserved for the gateway
gateway_ip = "${cidrhost(var.zt_cidr, 1)}" # eg 10.96.0.1
}
resource "zerotier_network" "your_network" {
name = "your_network_name"
assignment_pool {
first = "${cidrhost(var.zt_cidr, 2)}" # eg 10.96.0.2
last = "${cidrhost(var.zt_cidr, -2)}" # eg 10.96.0.254
}
route {
target = "${var.zt_cidr}"
}
route {
target = "${var.other_network}"
via = "${local.gateway_ip}"
}
}
Then go ahead and make an API call on your gateway's provisioner to set the IP address manually. See below (auto-joining).
Best of all, you can specify rules just like in the web interface. You could even use a Terraform template_file
to insert variables.
# ztr.conf
# drop non-v4/v6/arp traffic
drop not ethertype ipv4 and not ethertype arp and not ethertype ipv6;
# disallow tcp connections except by specific grant in a capability break chr tcp_syn and not chr tcp_ack;
# allow ssh from some devices
cap ssh
id 1000
accept ipprotocol tcp and dport 22;
;
# allow everything else
accept;
resource "zerotier_network" "your_network" {
name = "your_network_name"
assignment_pool {
cidr = "${var.zt_cidr}"
}
route {
target = "${var.zt_cidr}"
}
rules_source = "${file("ztr.conf")}"
}
Unfortunately, it is not possible for a machine to be added to a network without the machine itself reaching out to ZeroTier.
However, you can pre-approve a machine if you already know its Node ID. This is not the case for dynamically created machines like cloud instances. It is more useful for your developer machine, which you might want to give a bunch of capabilities and pre-approve so that when you do paste a network ID in, you don't have to use the web UI to do the rest.
Basic example to pre-approve:
resource "zerotier_member" "dev_machine" {
node_id = "..."
network_id = "${zerotier_network.net.id}"
name = "dev machine"
}
Full list of properties:
resource "zerotier_member" "hector" {
# required: the known id that a particular machine shows
# (e.g. in the Mac menu bar app, or the Windows tray, Linux CLI output)
node_id = "a1511e5bf5"
# required: the network id
network_id = "${zerotier_network.net.id}"
# the rest are optional
name = "hector"
description = "..."
authorized = true
# whether to show it in the list in the Web UI
hidden = false
# e.g.
# cap administrator
# id 1000
# accept;
# ;
capabilities = [ 1000 ]
# e.g.
# tag department
# id 2000
# enum 100 marketing
# enum 200 accounting
# ;
tags = {
"2000" = 100 # marketing
}
# default (false) means this member has a managed IP address automatically assigned.
# without ip_assignments being configured, the member won't have any managed IPs.
no_auto_assign_ips = false
# will happily override any auto-assigned v4 addresses (and v6 in some configurations)
ip_assignments = [
"10.0.96.15"
]
# not known whether this does anything or not
offline_notify_delay = 0
# see ZeroTier Manual section on L2/ethernet bridging
allow_ethernet_bridging = true
}
Things are simple when you already know your Node ID. A local-exec
provisioner
can be used to execute sudo zerotier-cli join [nwid]
when a network is
created, which will be auto-approved using a zerotier_member
resource. You
will have to type your password (once) during terraform apply
, or you will
have to apply as root already.
The provisioner should be defined on a null_resource
that is triggered when
the network ID changes. That way you can re-join by marking the null resource as
deleted, without deleting the entire network.
If you had another machine nearby (like a CI box), you could also run join
on
it using SSH or similar. Or just accept the one-off menial task.
resource "zerotier_network" "net" { ... }
resource "zerotier_member" "dev_machine" {
network_id = "${zerotier_network.net.id}"
node_id = "... (see above)"
name = "dev machine"
capabilities = [ 1000, 2000 ]
}
resource "null_resource" "joiner" {
triggers {
network_id = "${zerotier_network.net.id}"
}
provisioner "local-exec" {
command = "sudo zerotier-cli join ${zerotier_network.net.id}"
}
}
Using zerotier-cli join XXX
doesn't require an API key, but that member won't
be approved by default. On the other hand, the zerotier_member
resource cannot
force a machine to join, it can only (pre-)approve and (pre-)configure membership of
a machine whose Node ID is already known. This is not true of a dynamically
created instance on a cloud provider.
The solution is to pass in the key to a provisioner and use the ZeroTier REST API directly to do it from the instance itself. This is the basic pattern, and applies whether you're using Terraform provisioners, running Docker entrypoint scripts with environment variables, or running Ansible scripts (etc).
Any way you do it, you will need to have your ZT API key accessible to Terraform.
Provide the environment variable export TF_VAR_zerotier_api_key="..."
so you
can access the key outside the provider definition, and do something like this:
variable "zerotier_api_key" {}
provider "zerotier" {
api_key = "${var.zerotier_api_key}"
}
resource "zerotier_network" "example" {
# ...
}
You might then insert "${var.zerotier_api_key}"
into
a kubernetes_secret
resource, or an
aws_ssm_parameter
, or directly into a provisioner as a script
argument. To use a standard Terraform provisioner, do this:
resource "aws_instance" "web" {
provisioner "file" {
source = "join.sh"
destination = "/tmp/join.sh"
}
provisioner "remote-exec" {
inline = [
"sudo sh /tmp/join.sh ${var.zerotier_api_key} ${var.zerotier_network.example.id}"
]
}
}
Note the sudo
. join.sh
is like the following:
ZT_API_KEY="$1"
ZT_NET="$2"
# basically follow this guide
# https://www.zerotier.com/download.shtml
curl -s 'https://pgp.mit.edu/pks/lookup?op=get&search=0x1657198823E52A61' | gpg --import && \
if z=$(curl -s 'https://install.zerotier.com/' | gpg); then echo "$z" | sudo bash; fi
zerotier-cli join "$ZT_NET"
sleep 5
NODE_ID=$(zerotier-cli info | awk '{print $3}')
echo '{"config":{"authorized":true}}' | curl -X POST -H 'Authorization: Bearer $ZT_API_KEY' -d @- \
"https://my.zerotier.com/api/network/$ZT_NET/member/$NODE_ID"
You could even set a static IP there, by POSTing the following instead. This is
useful if you want the instance to act as a gateway with a known IP, like in the
multiple routes example above. Or any field from the ZeroTier API
Reference listing for POST /api/network/{networkId}/member/{nodeId}
.
{
"name": "a-single-tear",
"config": {
"authorized": true,
"ipAssignments": ["10.96.0.1"],
"capabilities": [ 1000, 2000 ],
"tags": [ [2000, 100] ]
}
}
If you:
- define a 'gateway' ec2 instance in a VPC
- make it auto-join/approve with a static IP address, as above
- add a managed ZT route to your VPC's CIDR via that static IP, using a
route
block
... then you're almost ready to replace a VPN gateway. This can be cheaper
and more flexible, and you can probably get by on a t2.nano
. The only
missing pieces are packet forwarding from ZT to VPC, and getting packets back
out.
It is preferable to set up your VPC route tables to route the ZeroTier CIDR
through your instance. If you have zero NAT, this means you will never have
any trouble with strange protocols, and you squeeze more performance out of
the t2.nano
you set up. To be fair, on a t2.nano
you are limited much
more by its limited link speed than anything else, and protocols that don't
support NAT are rare in primarily TCP/HTTP/ environments. NAT can be simpler
to set up if you have a lot of dynamically created subnets.
The main configuration difference of this approach from route table entries is that the
source IP is different for the security group rule evaluator. With plain
packet forwarding and a route table return, you need an ingress rule for your
zerotier CIDR on a service in your VPC. Say ingress tcp/80, 10.96.0.0/24
. This is
not more powerful, but equally as easy with Terraform. You can't control
where ZT assigns members within the assignment pools, and you would probably
regulate that with your ZT rules/capabilities anyway. With MASQUERADE, you
instead allow ingress from the gateway's security group.
Assuming the following:
networks:
zerotier = 10.96.0.0/24
aws vpc = 10.41.0.0/16
interfaces:
you = { zt0: 10.96.0.37 }
gateway = { zt0: 10.96.0.1, eth0: 10.41.1.15 }
ec2 in vpc = { eth0: 10.41.2.67 }
You'll need to enable Linux kernel IPV4 forwarding. Use your distro's version
of echo 1 > /proc/sys/net/ipv4/ip_forward
, and make it permanent by
editing/appending to /etc/sysctl.conf
. On Ubuntu, that's:
# requires sudo
# set up packet forwarding now
echo 1 > /proc/sys/net/ipv4/ip_forward
# make it permanent
echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf
It's a very good idea to have some FORWARD rules either way you do the routing, otherwise the gateway might be too useful as a nefarious pivot point into, inside or outbound from your VPC.
# requires sudo
iptables -F
# packets flow freely from zt to vpc
iptables -A FORWARD -i zt0 -o eth0 -s "$ZT_CIDR" -d "$VPC_CIDR" -j ACCEPT
# only allow stateful return in the other direction
# i.e. can't establish new outbound connections going the other way
iptables -A FORWARD -i eth0 -o zt0 -s "$VPC_CIDR" -d "$ZT_CIDR" -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -j REJECT
Load both of these scripts with your provisioner, whatever that may be, and run them as root.
You want packets to move like so:
in:
1. you.zt0(src=10.96.0.37, dest=10.41.2.67) => ZT => gateway.zt0
2. -> gateway.eth0(src=10.96.0.37, dest=10.41.2.67) => VPC normal => ec2.eth0
out:
3. ec2.eth0(src=10.41.2.67, dest=10.96.0.37) => VPC (through route table entry) => gateway.eth0
4. -> gateway.zt0(src=10.41.2.67, dest=10.96.0.37) => ZT => you.zt0
- #1 is satisfied because of the managed route, if your ZT flow rules allow it.
- #2 and #3 need an AWS security group on ec2 with inbound/outbound allowed to the ZeroTier CIDR
- #2 and #3 need EC2 IP source/dest check disabled on the gateway instance.
- #3 needs a a route table entry out of each subnet you want return traffic from, to the gateway instance.
- #4 is satisfied if your ZT flow rules allow it.
For packet forwarding, set source_dest_check = false
on the instance.
data "aws_route_table" "private" {
subnet_id = "..."
}
resource "aws_route" "zt_route" {
route_table_id = "${data.aws_route_table.private.id}"
# route all packets destined for zt network, send them through the gateway
destination_cidr_block = "${var.zt_cidr}"
instance_id = "${aws_instance.zt_gateway.id}"
}
You'll need a gateway security group with:
- All ingress UDP traffic on port 9993 allowed
- All egress allowed
- SSH ingress for provisioning
Any other ec2 instances you want to access from your ZT network will need:
- Ingress from your ZT CIDR for whatever ports you want
- Egress either everywhere or to ZT CIDR
The gateway behaves like a standard router, using iptables
MASQUERADE rules. 'You' sees exactly the same src,dest information on the packets; it looks like you are communicating directly with 10.41.2.67, but the 'ec2.eth0' interface sees packets coming from the gateway.
in:
1. you.zt0(src=10.96.0.37, dest=10.41.2.67) => ZT => gateway.zt0
2. -> gateway.eth0(src=10.41.1.15, dest=10.41.2.67) => VPC => ec2.eth0
out:
3. ec2.eth0(src=10.41.2.67, dest=10.41.1.15) => VPC => gateway.eth0
4. -> gateway.zt0(src=10.41.2.67, dest=10.96.0.37) => ZT => you.zt0
- #1 is satisfied because of the managed route, if your ZT flow rules allow it.
- #2 and #4 (NAT) need an
iptables
MASQUERADE rule on the gateway - #2 and #3 need an AWS security group on ec2 with inbound/outbound to allow the gateway's security group
- #4 is satisfied if your ZT flow rules allow it.
Append this to your FORWARD rules script above:
# zt0 is the zerotier virtual interface, eth0 is connected to the VPC
iptables -t nat -F
# make it look like packets are coming from the gateway, not a zerotier IP
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -j ACCEPT
You'll need a gateway security group with:
- All ingress UDP traffic on port 9993 allowed
- All egress allowed
- SSH ingress for provisioning
Any other ec2 instances you want to access from your ZT network will need:
- Ingress from your gateway's security group for whatever ports you want
- Egress either everywhere or to gateway
Terraform is able to import existing infrastructure. This allows you take resources you've created by some other means and bring it under Terraform management.
It is possible to import both networks and members state using this provider. Currently, Terraform features can only import the state information, and you still need to write the resource definition before it is able to import.
Given the following definition:
provider "zerotier" {
api_key = "..."
}
resource "zerotier_network" "your_network" {
name = "your_network_name"
}
resource "zerotier_member" "dev_machine" {
node_id = "..."
network_id = "${zerotier_network.your_network.id}"
}
You be able to import both the network
and the member
resources using the terraform import
command.
## Resource is identified by the network id by the API
terraform import zerotier_network.your_network ${NETWORK_ID}
## Resource is identified by the network id and the node id by the API
terraform import zerotier_member.dev_machine "${NETWORK_ID}-${NODE_ID}"
## Adjust the configuration until no change is planned
terraform plan