This exercise demonstrates some of the operational problems that applications in containers face.
Container schedulers (such as Kubernetes or Nomad) have a large number of responsibilities. Some of these responsibilities are inherent to the problem, whereas others are incidental. To understand which is which, we need to answer the question of "what do containers need in order run?"
A container needs:
- an image
- placement: CPU and memory on the host (ideally with good utilization)
- disk storage (optionally)
- networking: an IP address, possibly a public IP address, and port allocations or some kind of overlay networking
- service discovery: an ability to find its dependent services
For purpose of this workshop, we're not going to address the question of an image for the container, and we'll mostly put off disk storage (this is very platform-specific). We'll come back to the question of placement towards the end of the workshop. So let's examine the problems of networking.
Our demonstration application is a minimal HTTP API written in Python (using the Bottle framework). It takes a configuration file and serves one path: /user
. When the application gets a request on this path, it makes an outbound request to the GitHub API and fetches the URL associated with the avatar for the configured user, and this URL is returned in the JSON body.
We'll run the application in a Docker container.
First, let's check our config.json5. It should have been modified on login to include your GitHub account as the name
field.
{
name: "<your GitHub $ACCOUNT>",
token: "<GitHub OAuth token>",
host: "localhost",
port: 8080
}
# build and run the container using the default networking
$ docker build -t="workshop-$ACCOUNT" .
$ docker run -d -p 8080 --name "$ACCOUNT" "workshop-$ACCOUNT"
920dde8005ee
$ curl localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused
# what gives? let's check the container
$ docker ps -f name=$ACCOUNT
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
920dde8005ee workshop-tgross "python server.py" 1 minute ago Up 1 minute 0.0.0.0:32769->8080/tcp tgross
# ah we have NAT, so we need to change the address
$ curl localhost:32769
curl: (52) Empty reply from server
# our application isn't listening on the same "localhost"!
# it has a separate network!
# remove our previous container
$ cleanup
# run the same container with host networking
$ docker run -d -p 8080 --net=host --name "$ACCOUNT" "workshop-$ACCOUNT"
ab4234a46eff
# let's make sure it's up
$ docker ps -af name=$ACCOUNT
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ab4234a46eff workshop-tgross "python server.py" 4 seconds ago Exited (1) 3 seconds ago tgross
# why did it crash?
# note: yours might not have crashed if you were fastest on your VM
# with --net=host we share the network namespace with everyone
# else on the same VM, so we can have port conflicts
$ docker logs $ACCOUNT
Bottle v0.12.13 server starting up (using WSGIRefServer())...
Listening on http://localhost:8080/
Hit Ctrl-C to quit.
Traceback (most recent call last):
File "server.py", line 28, in <module>
run(host=app.config['host'], port=app.config['port'])
File "/usr/local/lib/python3.6/site-packages/bottle.py", line 3127, in run
server.run(app)
File "/usr/local/lib/python3.6/site-packages/bottle.py", line 2781, in run
srv = make_server(self.host, self.port, app, server_cls, handler_cls)
File "/usr/local/lib/python3.6/wsgiref/simple_server.py", line 153, in make_server
server = server_class((host, port), handler_class)
File "/usr/local/lib/python3.6/socketserver.py", line 453, in __init__
self.server_bind()
File "/usr/local/lib/python3.6/wsgiref/simple_server.py", line 50, in server_bind
HTTPServer.server_bind(self)
File "/usr/local/lib/python3.6/http/server.py", line 136, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/local/lib/python3.6/socketserver.py", line 467, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
Edit our config.json5 as follows:
{
name: "<your GitHub $ACCOUNT>",
token: "<GitHub OAuth token>",
host: "localhost",
port: <fill in your $PORT>
}
# remove our previous container
$ cleanup
# rebuild with the new config file
$ docker build -t="workshop-$ACCOUNT" .
# run the container again
$ docker run -d -p ${PORT}:${PORT} --net=host --name "$ACCOUNT" "workshop-$ACCOUNT"
eb4234756abc
$ docker ps -f name=$ACCOUNT
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
eb4234756abc workshop-tgross "python server.py" 1 minute ago Up 1 minute 0.0.0.0:8080->8080/tcp tgross
$ curl "localhost:${PORT}"
<div>
<h2>tgross</h2>
<div>
<img src=https://avatars3.githubusercontent.com/u/1409219?v=4 height="200", width="200"/>
</div>
</div>
Note that this requires cooperation of the application to accept an assigned dynamic port, but it avoids all overhead associated with an overlay networking solution or NAT. Note that in typical cloud environments you already have significant performance hits in networking due to hardware virtualization (+80% latency in a 2014 report from IBM (PDF)!), so adding a similar level of overhead on top of that is an extra hit. Host networking is suitable if you don't need multi-tenant safety.
We still haven't seen what this looks like, and we bind to localhost. So let's update our configuration to publish the API on the Internet.
Edit our config.json5 as follows:
{
name: "<your GitHub $ACCOUNT>",
token: "<GitHub OAuth token>",
host: "<fill in your $PUBLIC_IP",
port: <fill in your $PORT>
}
# remove our previous container
$ cleanup
# rebuild with the new config file
$ docker build -t="workshop-$ACCOUNT" .
# run the container again
$ docker run -d -p ${PORT}:${PORT} --net=host --name "$ACCOUNT" "workshop-$ACCOUNT"
14521a456adf
Now you should be able to reach your container from your browser, and fetch your own avatar. The URL http://${PUBLIC_IP}:${PORT}/user
will reach the JSON API, whereas the URL http://${PUBLIC_IP}:${PORT}/
will reach a simple web page that shows the avatar and your name.
A third option is to use an overlay networking solution -- which is a class of solutions that give each container its own network stack and IP address on the LAN. Some implementations:
- Docker overlay networking: go see Jerome's workshop at 2:30pm!
- Calico: based on BGP (strictly speaking not "overlay")
- Flannel: VXLAN or platform-specific backends (ex. AWS VPC)
- Triton: uses VXLAN for routing (demo Consul servers)
Our demo Consul servers are running as bare-metal containers on Triton, where each container gets a virtual NIC on a layer 2 private VLAN, and an optional VNIC on public routable network:
$ docker exec -it workshop_consul_1 ip addr
1: lo: <LOOPBACK,MULTICAST,UP> mtu 8232
link/loopback 00:00:00:00:00:00
inet 127.0.0.1/8 scope host dynamic
inet6 ::1/128 scope host dynamic
2: eth0: <BROADCAST,MULTICAST,UP> mtu 8500 qlen 1
link/ether 90:b8:d0:e5:d2:0c
inet 192.168.129.110/22 scope site dynamic
inet6 fe80::92b8:d0ff:fee5:d20c/10 scope link dynamic
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qlen 1
link/ether 90:b8:d0:23:cc:0a
inet 72.2.113.151/22 scope global dynamic
inet6 fe80::92b8:d0ff:fe23:cc0a/10 scope link dynamic
Most of the major schedulers expect containers to have their own IP address. Kubernetes expects each pod of containers to have its own IP address which is shared among containers in the pod. Containers in the pod communicate with each other over localhost. Kubernetes provides CNI plugins for solutios like Flannel and Calico.
Each platform implements this differently; value is that IP address you see in the container is the IP that other containers in the data center would use to connect it.