cni: use tmpfs location for ipam plugin (#24650)

When a Nomad host reboots, the network namespace files in the tmpfs in
`/var/run` are wiped out. So when we restore allocations after a host reboot, we
need to be able to restore both the network namespace and the network
configuration. But because the netns is newly created and we need to run the CNI
plugins again, this create potential conflicts with the IPAM plugin which has
written state to persistent disk at `/var/lib/cni`. These IPs aren't the ones
advertised to Consul, so there's no particular reason to keep them around after
a host reboot because all virtual interfaces need to be recreated too.

Reconfigure the CNI bridge configuration to use `/var/run/cni` as its state
directory. We already expect this location to be created by CNI because the
netns files are hard-coded to be created there too in `libcni`.

Note this does not fix the problem described for Docker in #24292 because that
appears to be related to the netns itself being restored unexpectedly from
Docker's state.

Ref: #24292 (comment)
Ref: https://www.cni.dev/plugins/current/ipam/host-local/#files

.changelog/24650.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+networking: use a tmpfs location for the state of CNI IPAM plugin used by bridge mode, to fix a bug where allocations would fail to restore after host reboot
+```

client/allocrunner/cni/bridge.go

@@ -58,9 +58,10 @@ func NewNomadBridgeConflist(conf NomadBridgeConfig) Conflist {
 			ForceAddress: true,
 			HairpinMode:  conf.HairpinMode,
 			Ipam: IPAM{
-				Type:   "host-local",
-				Ranges: ipRanges,
-				Routes: ipRoutes,
+				Type:    "host-local",
+				Ranges:  ipRanges,
+				Routes:  ipRoutes,
+				DataDir: "/var/run/cni",
 			},
 		},
 		Firewall{

client/allocrunner/cni/plugins.go

@@ -20,9 +20,10 @@ type Bridge struct {
 	Ipam         IPAM   `json:"ipam"`
 }
 type IPAM struct {
-	Type   string    `json:"type"`
-	Ranges [][]Range `json:"ranges"`
-	Routes []Route   `json:"routes"`
+	Type    string    `json:"type"`
+	Ranges  [][]Range `json:"ranges"`
+	Routes  []Route   `json:"routes"`
+	DataDir string    `json:"dataDir"`
 }
 type Range struct {
 	Subnet string `json:"subnet"`

client/allocrunner/test_fixtures/bad_input.conflist.json

@@ -25,7 +25,8 @@
 					{
 						"dst": "0.0.0.0/0"
 					}
-				]
+				],
+				"dataDir": "/var/run/cni"
 			}
 		},
 		{

client/allocrunner/test_fixtures/consul-cni.conflist.json

@@ -25,7 +25,8 @@
 					{
 						"dst": "0.0.0.0/0"
 					}
-				]
+				],
+				"dataDir": "/var/run/cni"
 			}
 		},
 		{

client/allocrunner/test_fixtures/empty.conflist.json

@@ -25,7 +25,8 @@
 					{
 						"dst": "0.0.0.0/0"
 					}
-				]
+				],
+				"dataDir": "/var/run/cni"
 			}
 		},
 		{

client/allocrunner/test_fixtures/hairpin.conflist.json

@@ -25,7 +25,8 @@
 					{
 						"dst": "0.0.0.0/0"
 					}
-				]
+				],
+				"dataDir": "/var/run/cni"
 			}
 		},
 		{

client/allocrunner/test_fixtures/ipv6.conflist.json

@@ -33,7 +33,8 @@
 					{
 						"dst": "::/0"
 					}
-				]
+				],
+				"dataDir": "/var/run/cni"
 			}
 		},
 		{