Add support for multiple network stanzas

[th0m]

Proposal

Sorry if I missed an existing issue for this, I have looked but couldn't find any.
I'd like to be able to define more than one network in my Nomad job file.
This could look like the following:

manynetworks.hcl

job "manynetworks" {                                                              
  datacenters = ["dc1"]                                                      
                                                                               
  group "manynetworks-group" {                                                    
    networks {       
      network {
        name = "vlan1"                                                           
        mode = "cni/vlan1"
      }
      network {
        name = "vlan2"                                                           
        mode = "cni/vlan2"
      }                                                   
    }
[...]

Use-cases

I have software that listens on multiple interfaces in different vlans.
I am using CNI to set up those interfaces but currently I am limited to a single interface.

/etc/cni/net.d/vlan1.conflist

{
    "cniVersion": "0.4.0",
    "name": "vlan1",
    "plugins" : [
        {
            "type": "vlan",
            "master": "eth0",
            "vlanId": 1,
            "ipam": {
                "type": "static",
                "addresses": [
                    {
                        "address": "10.0.1.1/24"
                    }
                ]
            },
         },
}

/etc/cni/net.d/vlan2.conflist

{
    "cniVersion": "0.4.0",
    "name": "vlan2",
    "plugins" : [
        {
            "type": "vlan",
            "master": "eth0",
            "vlanId": 2,
            "ipam": {
                "type": "static",
                "addresses": [
                    {
                        "address": "10.0.2.1/24"
                    }
                ]
            },
         },
}

The end result I am looking for in the container is

$ ip -d a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0.1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:9e:9e:dc:28:2f brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65535 
    vlan protocol 802.1Q id 1 <REORDER_HDR> numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 10.0.1.1/24 scope global eth0.1
       valid_lft forever preferred_lft forever
    inet6 fe80::9e:9eff:fedc:282f/64 scope link 
       valid_lft forever preferred_lft forever
3: eth0.2@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:9e:9e:dc:28:2e brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65535 
    vlan protocol 802.1Q id 2 <REORDER_HDR> numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 10.0.2.1/24 scope global eth0.2
       valid_lft forever preferred_lft forever
    inet6 fe80::9e:9eff:fedc:282e/64 scope link 
       valid_lft forever preferred_lft forever

Attempted Solutions

I have tried to solve this on CNI end by adding more than one plugin to my network.

/etc/cni/net.d/vlan1and2.conflist

{
    "cniVersion": "0.4.0",
    "name": "vlan1and2",
    "plugins" : [
        {
            "type": "vlan",
            "master": "eth0",
            "vlanId": 1,
            "ipam": {
                "type": "static",
                "addresses": [
                    {
                        "address": "10.0.1.1/21"
                    }
                ]
            },
         },
        {
            "type": "vlan",
            "master": "eth0",
            "vlanId": 2,
            "ipam": {
                "type": "static",
                "addresses": [
                    {
                        "address": "10.0.2.1/21"
                    }
                ]
            },
         },
    ]
}

But this does not work because the resulting ifName interface name is eth0 for both and the second interface therefore fails to create.
This is due to the fact the index following CNIInterfacePrefix is per CNI network, not per plugin.
https://github.com/hashicorp/nomad/blob/main/client/config/config.go#L242-L243

Alternatively I can use the macvlan CNI plugin and defer creating the vlan interfaces inside the container but that requires CAP_NET_ADMIN privileges which I'd rather drop.

Also alternatively I could spin up as many tasks as I need vlans but I'd rather keep everything in one as it makes everything simpler.

Let me know if you are planning on supporting multiple network stanzas at some point or if you can think of another way to solve this.
Thank you!

[jrasell]

Hi @th0m and thanks so much for the information provided. I have been unable to find a workaround during my investigations and therefore will accept this as an enhancement to be roadmapped.

Interestingly, the network block within the Nomad API is a list which means this example jobfile is valid and registers correctly with Nomad.

Example Job

job "example" {
  datacenters = ["dc1"]

  group "vlan_cni" {
    network {
      mode = "cni/vlan1"
    }
    network {
      mode = "cni/vlan2"
    }

    task "vlan_cni" {
      driver = "docker"

      config {
        image   = "praqma/network-multitool"
        command = "sleep"
        args    = ["3000s"]
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

When creating the network, however, the first element is only ever interrogated which results in only vlan1 being added to the container network.

For future readers the two following CNI configuration files can be placed within /opt/cni/config on the Nomad Vagrant machine for testing running Nomad via sudo nomad agent -dev.

vlan1.conflist

{
	"cniVersion": "0.4.0",
	"name": "vlan1",
	"plugins": [{
		"type": "vlan",
		"master": "eth0",
		"vlanId": 1,
		"ipam": {
			"type": "static",
			"addresses": [{
				"address": "10.0.1.1/24"
			}]
		}
	}]
}

vlan2.conflist

{
	"cniVersion": "0.4.0",
	"name": "vlan2",
	"plugins": [{
		"type": "vlan",
		"master": "eth0",
		"vlanId": 2,
		"ipam": {
			"type": "static",
			"addresses": [{
				"address": "10.0.2.1/24"
			}]
		}
	}]
}

[th0m]

Thank you @jrasell.

[th0m]

Just a note here that the multi-interface in a pod can be done in Kubernetes by using Multus CNI that is essentially a meta-plugin calling other CNI plugins.
That CNI plugin is Kubernetes specific so not something we can use with Nomad.

Anyway, I do believe the right place and way to implement support in Nomad is what @jrasell described in their comment.

[th0m]

Hi @jrasell, I just noticed this was removed from "Needs Roadmapping", does this mean this will not be worked on internally?
Thank you!

[tgross]

Hi @th0m! The issue was moved to our internal "unified backlog" (which unfortunately isn't public), but just FYI it's not on our highest priority queue for near-term work.

[th0m]

Hi @tgross, sounds good, thanks for the quick response!

[th0m]

Hi @tgross and @jrasell I was wondering if you had this feature request on your radar? We are getting more use cases for it cropping up internally.
Thank

[th0m]

Hi, just wondering if this is something that will be supported upstream at some point? We have an internal patch for it that is not really upstreamable and we'd love to replace it with an upstream solution.

[davidlublink]

Hello @th0m ,

I also need a container in nomad with 2 interfaces on different networks. In my case I am using ipvlan and not vlan.

Here is the workaround I am working on, first, instead of having multiple network stanzas in my nomad job definition, I have multiple interfaces defined in my /opt/cni/somefile.confist :

{
     "cniVersion": "0.4.0",
          "name": "somenetwork",
          "plugins": [
          {
               "type": "ipvlan",
               "master": "enp0s25",
               "ipam": {
                    "type": "static",
                    "addresses": 
                         [
                    {
                         "address": "10.99.0.5/24",
                         "gateway": "10.99.0.1"
                    }],

                    "routes" : [
                         { "dst": "0.0.0.0/0" }
                    ],
                    "dns": {
                         "nameservers" : ["8.8.8.8"],
                         "domain": "example.com",
                         "search": [ "example.com" ]
                    }
               }
          },
          {
               "type": "ipvlan1",
               "master": "enp0s25",
               "ipam": {
                    "type": "static",
                    "addresses": 
                         [
                    {
                         "address": "192.168.99.5/24",
                         "gateway": "192.168.99.1"
                    }],

                    "routes" : [
                         { "dst": "0.0.0.0/0" }
                    ],
                    "dns": {
                         "nameservers" : ["8.8.8.8"],
                         "domain": "example.com",
                         "search": [ "example.com" ]
                    }
               }
          },
          {
               "type": "portmap",
               "capabilities": { "portMappings": true },
               "snat": true
          }
     ]
}

Now it turns out that the 'type' in this definition is directly mapped to /opt/cni/bin/type, eg we invoke /opt/cni/bin/ipvlan and /opt/cni/bin/ipvlan1 executable when "CNI" sets up the network space for our container.

The contents of /opt/cni/bin/ipvlan are an elf binary I downloaded during setup.

The contents of ipvlan1 are the following :

#!/bin/bash
CNI_IFNAME=eth1
exec /opt/cni/bin/ipvlan

Turns out Nomad actually handles multiple interfaces, except it always sends 'eth0' as the CNI_IFNAME when invoking the CNI scripts. So I added this intermediate that overrides it and sets it to eth1 instead.

With this, I was able to successfully start in Nomad 2 interfaces :

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:1e:37:46:cc:72 brd ff:ff:ff:ff:ff:ff
    inet 10.99.0.5/24 brd 10.99.0.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:1e:37:46:cc:72 brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.5/24 brd 192.168.99.255 scope global eth1
       valid_lft forever preferred_lft forever

In my test lab, it works in that I see both interfaces. I have not tested any traffic yet. You'll notice that they have the same MAC, I assume this is because they are on the same bridge in my setup.

Can you give this work around a try and see if it works for your case?

David

// failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to configure network: plugin type="ipvlan" failed (add): failed to rename ipvlan to "eth0": file exists

[lgfa29]

Hi all 👋

I don't have any updates on this, but just want to note that this feature request has also been proposed in #13824.

[the-maldridge]

This seems like one of the cleaner solutions to the problems I raised in #13824 @lgfa29; where jobs could be launched into the default network when a group is first using nomad, and then later more advanced network solutions could be enabled and gradually adopted. Other solutions seem to involve a flag day or other big-bang cutover, and having run those at several companies now they're pretty painful.

[lgfa29]

Good point! It's a useful feature on it's on, but it can also be helpful on progressive adoption of new network configs, which, as you mentioned, is always nicer than having to do it all at once.

[MikeZappa87]

Hello, I am one of the contained/cni maintainers. I have started work on executing multiple cni network configurations in go-cni and containerd (cri plugin). This might be of use for Nomad.

[nathanaelle]

Hello, I am one of the contained/cni maintainers. I have started work on executing multiple cni network configurations in go-cni and containerd (cri plugin). This might be of use for Nomad.

@MikeZappa87 is there a repository with some piece of code ?
does this repository accept external contributions ?

[jbilbro]

Any updates on this feature request?

[lgfa29]

Hi @jbilbro 👋

No updates yet, we will let you all know in case there are updates.

If you haven't already, I would suggest giving this issue a 👍 to help us gauge interest.

Thank you!