This is a really simple application I wrote over holidays a year ago (12/17) that details my experiences and feedback with istio. To be clear, its a really basic NodeJS application that i used here but more importantly, it covers the main sections of Istio that i was seeking to understand better (if even just as a helloworld).
I do know isito has the "bookinfo" application but the best way i understand something is to rewrite sections and only those sections from the ground up.
- 11/14/24: Istio 1.24.0
- 04/03/24: Istio 1.21.0
- 09/05/22: Istio 1.15.0
- 03/19/21: Istio 1.9.1
- 12/21/20: Istio 1.8.0
- 09/22/20: Istio 1.7.2
- 04/28/20: Istio 1.5.2
- 10/12/19: Istio 1.3.2
- 03/10/19: Istio 1.1.0
- 01/09/19: Istio 1.0.5
- Prior Istio Versions
- Basic istio Installation on Google Kubernetes Engine
- Grafana, Prometheus, Kiali, Jaeger
- Route Control
- Canary Deployments with VirtualService
- Destination Rules
- Egress Rules
- Egress Gateway
- WebAssembly
- LUA HttpFilter
- Authorization
- JWT Authentication an Authorization
- Service to Service Authentication Policy
- Internal LoadBalancer (GCP)
- Mixer Out of Process Authorization Adapter
- Access GCE MetadataServer
You can also find info about istio+external authorization server here:
NodeJS in a Dockerfile...something really minimal. You can find the entire source under the 'nodeapp' folder in this repo.
The endpoints on this app are as such:
/: Does nothing; (source)/varz: Returns all the environment variables on the current Pod (source)/version: Returns just the "process.env.VER" variable that was set on the Deployment (source)/backend: Return the nodename, pod name. Designed to only get called as if the applciation running is abackend(source)/hostz: Does a DNS SRV lookup for thebackendand makes an http call to its/backend, endpoint (source)/requestz: Makes an HTTP fetch for several external URLs (used to show egress rules) (source)/headerz: Displays inbound headers (source)/remote: Access/backendwhile deployed in a remote istio cluster (source)
I build and uploaded this app to dockerhub at
docker.io/salrashid123/istioinit:1
docker.io/salrashid123/istioinit:2
(basically, they're both the same application but each has an environment variable that signifies which 'version; they represent. The version information for each image is returned by the /version endpoint)
You're also free to build and push these images directly:
docker build --build-arg VER=1 -t your_dockerhub_id/istioinit:1 .
docker build --build-arg VER=2 -t your_dockerhub_id/istioinit:2 .
docker push your_dockerhub_id/istioinit:1
docker push your_dockerhub_id/istioinit:2To give you a sense of the differences between a regular GKE specification yaml vs. one modified for istio, you can compare:
- all-istio.yaml vs all-gke.yaml (review Ingress config, etc)
Use Minikube locally..its easier but you can't really test the internal loadbalancer excercise.
note, for GKE, the following cluster is setup with a aliasIPs (--enable-ip-alias )
We will be installing istio with istioctl
minikube start --driver=kvm2 --cpus=4 --kubernetes-version=v1.28 --host-only-cidr 192.168.39.1/24
minikube addons enable metallb
## get the IP, for me it was the following
$ minikube ip
192.168.39.1
## setup a loadbalancer metallb, enter the ip range shown below
minikube addons configure metallb
# -- Enter Load Balancer Start IP: 192.168.39.104
# -- Enter Load Balancer End IP: 192.168.39.110
### in a new window
minikube dashboard
## for GKE
# gcloud container clusters create cluster-1 --machine-type "n2-standard-4" --zone us-central1-a --num-nodes 4 --enable-ip-alias -q
# gcloud container clusters get-credentials cluster-1 --zone us-central1-a
# kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value core/account)
kubectl create ns istio-system
export ISTIO_VERSION=1.24.0
export ISTIO_VERSION_MINOR=1.24
wget -P /tmp/ https://github.com/istio/istio/releases/download/$ISTIO_VERSION/istio-$ISTIO_VERSION-linux-amd64.tar.gz
tar xvf /tmp/istio-$ISTIO_VERSION-linux-amd64.tar.gz -C /tmp/
rm /tmp/istio-$ISTIO_VERSION-linux-amd64.tar.gz
export PATH=/tmp/istio-$ISTIO_VERSION/bin:$PATH
istioctl install --set profile=demo \
--set meshConfig.enableAutoMtls=true \
--set values.gateways.istio-ingressgateway.runAsRoot=true \
--set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY \
--set meshConfig.defaultConfig.gatewayTopology.forwardClientCertDetails=SANITIZE_SET
### GKE https://github.com/GoogleCloudPlatform/istio-samples/tree/master/internal-load-balancer
### add to istioctl command above -f overlay-istio-gateway.yaml
$ istioctl profile dump --config-path components.ingressGateways demo
$ istioctl profile dump --config-path values.gateways.istio-ingressgateway demo
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/prometheus.yaml
sleep 20
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/jaeger.yaml
kubectl label namespace default istio-injection=enabledWait maybe 2 to 3 minutes and make sure all the Deployments are live:
- For reference, here are the Istio operator installation options
Verify this step by making sure all the Deployments are Available.
$ kubectl get no,po,rc,svc,ing,deployment -n istio-system
NAME STATUS ROLES AGE VERSION
node/minikube Ready control-plane 6m55s v1.28.13
NAME READY STATUS RESTARTS AGE
pod/grafana-6f7c87f789-8ptwb 1/1 Running 0 73s
pod/istio-egressgateway-5686766648-wrpj9 1/1 Running 0 5m44s
pod/istio-ingressgateway-7647b7fcfc-8hxhv 1/1 Running 0 5m43s
pod/istiod-5fbd9b746d-h7mjc 1/1 Running 0 6m4s
pod/jaeger-6d58dbf847-492js 1/1 Running 0 72s
pod/kiali-7d57f454c-p7wcf 1/1 Running 0 74s
pod/prometheus-858b48bf9b-r82nk 2/2 Running 0 95s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/grafana ClusterIP 10.107.88.114 <none> 3000/TCP 73s
service/istio-egressgateway ClusterIP 10.105.163.55 <none> 80/TCP,443/TCP 5m44s
service/istio-ingressgateway LoadBalancer 10.101.132.30 192.168.39.104 15021:32216/TCP,80:32466/TCP,443:30304/TCP,31400:30523/TCP,15443:32664/TCP 5m44s
service/istiod ClusterIP 10.110.231.171 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 6m4s
service/jaeger-collector ClusterIP 10.104.227.189 <none> 14268/TCP,14250/TCP,9411/TCP,4317/TCP,4318/TCP 72s
service/kiali ClusterIP 10.108.203.182 <none> 20001/TCP,9090/TCP 74s
service/prometheus ClusterIP 10.100.159.4 <none> 9090/TCP 95s
service/tracing ClusterIP 10.108.168.98 <none> 80/TCP,16685/TCP 72s
service/zipkin ClusterIP 10.96.117.238 <none> 9411/TCP 72s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 73s
deployment.apps/istio-egressgateway 1/1 1 1 5m44s
deployment.apps/istio-ingressgateway 1/1 1 1 5m44s
deployment.apps/istiod 1/1 1 1 6m4s
deployment.apps/jaeger 1/1 1 1 72s
deployment.apps/kiali 1/1 1 1 74s
deployment.apps/prometheus 1/1 1 1 95sRun
$ kubectl get svc istio-ingressgateway -n istio-system
export GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $GATEWAY_IPOpen up several new shell windows and type in one line into each:
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app.kubernetes.io/name=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000
kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 16686:16686
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app.kubernetes.io/name=kiali -o jsonpath='{.items[0].metadata.name}') 20001:20001Open up a browser (4 tabs) and go to:
- Kiali http://localhost:20001/kiali (username: admin, password: admin)
- Grafana http://localhost:3000/dashboards
- Jaeger http://localhost:16686
The default all-istio.yaml runs:
- Ingress with SSL
- Deployments:
-
- myapp-v1: 1 replica
-
- myapp-v2: 1 replica
-
- be-v1: 1 replicas
-
- be-v2: 1 replicas
basically, a default frontend-backend scheme with one replicas for each v1 and v2 versions.
Note: the default yaml pulls and run my dockerhub image- feel free to change this if you want.
kubectl apply -f all-istio.yaml
kubectl apply -f istio-lb-certs.yamlNow enable the ingress gateway for both external and internal loadbalancer traffic on only port :443:
kubectl apply -f istio-ingress-gateway.yaml -f istio-ingress-ilbgateway.yaml
kubectl apply -f istio-fev1-bev1.yamlWait until the deployments complete:
$ kubectl get po,deployments,svc,ing
NAME READY STATUS RESTARTS AGE
pod/be-v1-965f5d5d4-2vb46 2/2 Running 0 85s
pod/be-v2-859c86c476-vfpmp 2/2 Running 0 85s
pod/myapp-v1-f65cf8f46-gvccx 2/2 Running 0 85s
pod/myapp-v2-56f6698cd4-fjh2f 2/2 Running 0 85s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/be-v1 1/1 1 1 85s
deployment.apps/be-v2 1/1 1 1 85s
deployment.apps/myapp-v1 1/1 1 1 85s
deployment.apps/myapp-v2 1/1 1 1 85s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/be ClusterIP 10.101.138.254 <none> 8080/TCP 85s
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10m
service/myapp ClusterIP 10.101.150.123 <none> 8080/TCP 85sNotice that each pod has two containers: one is from isto, the other is the applicaiton itself (this is because we have automatic sidecar injection enabled on the default namespace).
Also note that in all-istio.yaml we did not define an Ingress object though we've defined a TLS secret with a very specific metadata name: istio-ingressgateway-certs. That is a special name for a secret that is used by Istio to setup its own ingress gateway:
Note the istio-ingress-gateway secret specifies the Ingress cert to use (the specific metadata name is special and is required)
apiVersion: v1
data:
tls.crt: _redacted_
tls.key: _redacted_
kind: Secret
metadata:
name: istio-ingressgateway-certs
namespace: istio-system
type: kubernetes.io/tlsRemember we've acquired the $GATEWAY_IP earlier:
export GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $GATEWAY_IPyou may need to restart the ingress pod if the certs they used didn't pickup
INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl delete po/$INGRESS_POD_NAME -n istio-systemThis section shows basic user->frontend traffic and see the topology and telemetry in the Kiali and Grafana consoles:
So...lets send traffic with the ip to the /versions on the frontend
for i in {1..1000}; do curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/version; sleep 1; doneYou should see a sequence of 1's indicating the version of the frontend you just hit
111111111111111111111111111111111
(source: /version endpoint)
You should also see on kiali just traffic from ingress -> fe:v1
and in grafana:
Now the next step in th exercise:
to send requests to user-->frontend--> backend; we'll use the /hostz endpoint to do that. Remember, the /hostz endpoint takes a frontend request, sends it to the backend which inturn echos back the podName the backend runs as. The entire response is then returned to the user. This is just a way to show the which backend host processed the requests.
(note i'm using jq utility to parse JSON)
for i in {1..1000}; do curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/hostz | jq '.[0].body'; sleep 1; doneyou should see output indicating traffic from the v1 backend verison: be-v1-*. Thats what we expect since our original rule sets defines only fe:v1 and be:v1 as valid targets.
$ for i in {1..1000}; do curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"Note both Kiali and Grafana shows both frontend and backend service telemetry and traffic to be:v1
This section details how to selectively send traffic to specific service versions and control traffic routing.
In this sequence, we will setup a routecontrol to:
- Send all traffic to
myapp:v1. - traffic from
myapp:v1can only go tobe:v2
Basically, this is a convoluted way to send traffic from fe:v1-> be:v2 even if all services and versions are running.
The yaml on istio-fev1-bev2.yaml would direct inbound traffic for myapp:v1 to go to be:v2 based on the sourceLabels:. The snippet for this config is:
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: be-virtualservice
spec:
gateways:
- mesh
hosts:
- be
http:
- match:
- sourceLabels:
app: myapp
version: v1
route:
- destination:
host: be
subset: v2
weight: 100
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: be-destination
spec:
host: be
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
loadBalancer:
simple: ROUND_ROBIN
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2So lets apply the config with kubectl:
kubectl replace -f istio-fev1-bev2.yamlAfter sending traffic, check which backend system was called by invoking /hostz endpoint on the frontend.
What the /hostz endpoint does is takes a users request to fe-* and targets any be-* that is valid. Since we only have fe-v1 instances running and the fact we setup a rule such that only traffic from fe:v1 can go to be:v2, all the traffic outbound for be-* must terminate at a be-v2:
$ for i in {1..1000}; do curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"and on the frontend version is always one.
for i in {1..100}; do curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/version; sleep 1; done
11111111111111111111111111111Note the traffic to be-v1 is 0 while there is a non-zero traffic to be-v2 from fe-v1:
Look at the incoming requests by source graph:
If we now overlay rules that direct traffic allow interleaved fe(v1|v2) -> be(v1|v2) we expect to see requests to both frontend v1 and backend
kubectl replace -f istio-fev1v2-bev1v2.yamlthen frontend is both v1 and v2:
for i in {1..1000}; do curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/version; sleep 1; done
111211112122211121212211122211and backend is responses comes from both be-v1 and be-v2
$ for i in {1..1000}; do curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
Now lets setup a more selective route based on a specific path in the URI:
- The rule we're defining is: "First route requests to myapp where
path=/versionto only go to thev1set"...if there is no match, fall back to the default routes where you send20%traffic tov1and80%traffic tov2
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: myapp-virtualservice
spec:
hosts:
- "*"
gateways:
- my-gateway
http:
- match:
- uri:
exact: /version
route:
- destination:
host: myapp
subset: v1
- route:
- destination:
host: myapp
subset: v1
weight: 20
- destination:
host: myapp
subset: v2
weight: 80kubectl replace -f istio-route-version-fev1-bev1v2.yamlSo check all requests to /version are fe:v1
for i in {1..1000}; do curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/version; sleep 1; done
1111111111111111111You may have noted how the route to any other endpoint other than /version destination is weighted split and not delcared round robin (eg:)
- route:
- destination:
host: myapp
subset: v1
weight: 20
- destination:
host: myapp
subset: v2
weight: 80Anyway, now lets edit rule to and change the prefix match to /xversion so the match doesn't apply. What we expect is a request to http://gateway_ip/version will go to v1 and v2 (since the path rule did not match and the split is the fallback rule.
kubectl replace -f istio-route-version-fev1-bev1v2.yaml
Observe the version of the frontend you're hitting:
for i in {1..1000}; do curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/version; sleep 1; done
2121212222222222222221122212211222222222222222What you're seeing is myapp-v1 now getting about 20% of the traffic while myapp-v2 gets 80% because the previous rule doens't match.
Undo that change /xversion --> /version and reapply to baseline:
kubectl replace -f istio-route-version-fev1-bev1v2.yamlYou can use this traffic distribuion mechanism to run canary deployments between released versions. For example, a rule like the following will split the traffic between v1|v2 at 80/20 which you can use to gradually roll traffic over to v2 by applying new percentage weights.
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: myapp-virtualservice
spec:
hosts:
- "*"
gateways:
- my-gateway
- my-gateway-ilb
http:
- route:
- destination:
host: myapp
subset: v1
weight: 80
- destination:
host: myapp
subset: v2
weight: 20Lets configure Destination rules such that all traffic from myapp-v1 round-robins to both version of the backend.
First lets force all gateway requests to go to v1 only:
on istio-fev1-bev1v2.yaml:
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: myapp-virtualservice
spec:
hosts:
- "*"
gateways:
- my-gateway
- my-gateway-ilb
http:
- route:
- destination:
host: myapp
subset: v1And where the backend trffic is split between be-v1 and be-v2 with a ROUND_ROBIN
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: be-destination
spec:
host: be
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
loadBalancer:
simple: ROUND_ROBIN
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2After you apply the rule,
kubectl replace -f istio-fev1-bev1v2.yamlyou'll see frontend request all going to fe-v1
for i in {1..1000}; do curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/version; sleep 1; done
11111111111111with backend requests coming from pretty much round robin
$ for i in {1..1000}; do curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"Now change the istio-fev1-bev1v2.yaml to RANDOM and see response is from v1 and v2 random:
$ for i in {1..1000}; do curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v2-859c86c476-vfpmp] node: [minikube]"
"pod: [be-v1-965f5d5d4-2vb46] node: [minikube]"The configuration here sets up an internal loadbalancer on GCP to access an exposed istio service.
THis only applies if you set up the overlay-istio-gateway.yaml operator during setup for GKE only
The config settings that enabled this during istio setup was done by an operator and annotation:
Specifically, we created a new ingressGateway and set its annotation to
cloud.google.com/load-balancer-type: "internal"
apiVersion: install.istio.io/v1
kind: IstioOperator
spec:
components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
service:
ports:
- name: https
port: 443
protocol: TCP
- name: istio-ilbgateway
enabled: true
k8s:
serviceAnnotations:
cloud.google.com/load-balancer-type: "internal"
service:
ports:
- port: 443
name: https
protocol: TCPWe did that duirng setup and later on, attached a Gateway to it which also exposed only :443
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
name: my-gateway-ilb
spec:
selector:
istio: istio-ilbgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "*"
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ilbgateway-certs/tls.crt
privateKey: /etc/istio/ilbgateway-certs/tls.key We also specified a VirtualService which selected these inbound gateways to the myapp service: This configuration was defined when we applied istio-fev1-bev1.yaml:
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: myapp-virtualservice
spec:
hosts:
- "*"
gateways:
- my-gateway
- my-gateway-ilb
http:
- route:
- destination:
host: myapp
subset: v1
weight: 100Note the gateways: entry in the VirtualService includes my-gateway-ilb which is what defines host:myapp, subset:v1 as a target for the ILB
gateways:
- my-gateway
- my-gateway-ilbAs mentioned above, we had to manually specify the port the ILB will listen on for traffic inbound to this service. \
Finally, the certficates Secret mounted at /etc/istio/ilbgateway-certs/ was specified this in the initial all-istio.yaml file:
apiVersion: v1
data:
tls.crt: LS0tLS1CR...
tls.key: LS0tLS1CR...
kind: Secret
metadata:
name: istio-ilbgateway-certs
namespace: istio-system
type: kubernetes.io/tlsNow that the service is setup, acquire the ILB IP allocated
export ILB_GATEWAY_IP=$(kubectl -n istio-system get service istio-ilbgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $ILB_GATEWAY_IP
Then from a GCE VM in the same VPC, send some traffic over on the internal address
$ curl -vk https://10.128.0.122/
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; O=Google; OU=Enterprise; CN=gke.default.svc.cluster.local
* start date: Dec 24 18:17:46 2017 GMT
* expire date: Jun 11 18:17:46 2020 GMT
* issuer: C=US; ST=California; L=Mountain View; O=Google; OU=Enterprise; CN=TestCAforESO
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/2
> Host: 10.128.0.122
> User-Agent: curl/7.64.0
> Accept: */*
< HTTP/2 200
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 19
< etag: W/"13-tsbq4e7agwVV6r9iE+Lb/lLwlzw"
< date: Tue, 22 Sep 2020 16:24:55 GMT
< x-envoy-upstream-service-time: 3
< server: istio-envoy
Hello from Express!- The Kiali console should show traffic from both gateways (if you recently sent traffic in externally and internally):
By default, istio blocks the cluster from making outbound requests. There are several options to allow your service to connect externally:
- Egress Rules
- Egress Gateway
- Setting
global.proxy.includeIPRanges
Egress rules prevent outbound calls from the server except with whitelisted addresses.
For example:
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
name: bbc-ext
spec:
hosts:
- www.bbc.com
ports:
- number: 80
name: http
protocol: HTTP
resolution: DNS
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
name: google-ext
spec:
hosts:
- www.google.com
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: google-ext
spec:
hosts:
- www.google.com
tls:
- match:
- port: 443
sni_hosts:
- www.google.com
route:
- destination:
host: www.google.com
port:
number: 443
weight: 100
Allows only http://www.bbc.com/* and https://www.google.com/*
To test the default policies, the /requestz endpoint tries to fetch the following URLs:
var urls = [
'https://www.google.com/robots.txt',
'http://www.bbc.com/robots.txt',
'http://www.google.com:443/robots.txt',
'https://www.cornell.edu/robots.txt',
'https://www.uwo.ca/robots.txt',
'http://www.yahoo.com/robots.txt'
]First make sure there is an inbound rule already running:
kubectl replace -f istio-fev1-bev1.yamlAnd that you're using REGISTRY_ONLY:
kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: REGISTRY_ONLY"- Without egress rule, requests will fail:
curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/requestz | jq '.'gives
[
{
"url": "https://www.google.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
},
{
"url": "http://www.google.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.bbc.com/robots.txt",
"body": "",
"statusCode": 502
},
{
"url": "https://www.cornell.edu/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.uwo.ca/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
},
{
"url": "https://www.yahoo.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
},
{
"url": "http://www.yahoo.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
]
Note: the
502response for thebbc.comentry is the actual denial rule from the istio-proxy (502->Bad Gateway)
then apply the egress policy which allows www.bbc.com:80 and www.google.com:443
kubectl apply -f istio-egress-rule.yamlgives
curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/requestz | jq '.'[
{
"url": "https://www.google.com/robots.txt",
"statusCode": 200
},
{
"url": "http://www.google.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.bbc.com/robots.txt",
"statusCode": 200
},
{
"url": "https://www.cornell.edu/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.uwo.ca/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.yahoo.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.yahoo.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
]
Notice that only one of the hosts worked over SSL worked
THe egress rule above initiates the proxied connection from each sidecar....but why not initiate the SSL connection from a set of bastion/egress
gateways we already setup? THis is where the Egress Gateway configurations come up but inorder to use this: The following configuration will allow egress traffic for www.yahoo.com via the gateway. See HTTPS Egress Gateway
So.. lets revert the config we setup above
kubectl delete -f istio-egress-rule.yamlthen lets apply the rule for the gateway:
kubectl apply -f istio-egress-gateway.yamlNotice the gateway TLS mode is PASSTHROUGH ("Note the PASSTHROUGH TLS mode which instructs the gateway to pass the ingress traffic AS IS, without terminating TLS.")
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
name: istio-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: 443
name: tls
protocol: TLS
hosts:
- www.yahoo.com
tls:
mode: PASSTHROUGHcurl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/requestz | jq '.'[
{
"url": "https://www.google.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.google.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.bbc.com/robots.txt",
"body": "",
"statusCode": 502
},
{
"url": "https://www.cornell.edu/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.uwo.ca/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.yahoo.com/robots.txt",
"statusCode": 200 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
},
{
"url": "http://www.yahoo.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
}
}
]
You can also tail the egress gateway logs:
$ kubectl logs -f --tail=0 -l istio=egressgateway -n istio-system
[2020-04-29T15:23:39.949Z] "- - -" 0 - "-" "-" 829 5706 144 - "-" "-" "-" "-" "72.30.35.10:443" outbound|443||www.yahoo.com 10.12.1.4:57332 10.12.1.4:443 10.12.2.10:41592 www.yahoo.com -
[2020-04-29T15:23:48.195Z] "- - -" 0 - "-" "-" 829 5722 138 - "-" "-" "-" "-" "98.138.219.231:443" outbound|443||www.yahoo.com 10.12.1.4:40632 10.12.1.4:443 10.12.2.10:41658 www.yahoo.com -In this mode, traffic exits the pod unencrypted but gets proxied via the gateway for an https destination. For this to work, traffic must originate from the pod unencrypted but specify the port as an SSL port. In current case, if you want to send traffic for https://www.yahoo.com/robots.txt, emit the request from the pod as http://www.yahoo.com:443/robots.txt. Note the traffic is http:// and the port is specified: :443
Ok, lets try it out, apply:
kubectl apply -f istio-egress-gateway-tls-origin.yamlThen notice just the last, unencrypted traffic to yahoo succeeds
curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/requestz | jq '.'
[
{
"url": "https://www.google.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
}
},
{
"url": "http://www.google.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
}
},
{
"url": "http://www.bbc.com/robots.txt",
"body": "",
"statusCode": 502
},
{
"url": "https://www.cornell.edu/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
},
{
"url": "https://www.uwo.ca/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
}
},
{
"url": "http://www.yahoo.com/robots.txt",
"statusCode": 200
},
{
"url": "https://www.yahoo.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
}
},
{
"url": "http://www.yahoo.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
}
}
]
You can also configure the global.proxy.includeIPRanges= variable to completely bypass the IP ranges for certain serivces. This setting is described under Calling external services directly and details the ranges that should get covered by the proxy. For GKE, you need to cover the subnets included and allocated:
The /metadata endpoint access the GCE metadata server and returns the current projectID. This endpoint makes three separate requests using the three formats I've see GCP client libraries use. (note: the hostnames are supposed to resolve to the link local IP address shown below)
app.get('/metadata', (request, response) => {
var resp_promises = []
var urls = [
'http://metadata.google.internal/computeMetadata/v1/project/project-id',
'http://metadata/computeMetadata/v1/project/project-id',
'http://169.254.169.254/computeMetadata/v1/project/project-id'
]So if you make an inital request, you'll see 404 errors from Envoy since we did not setup any rules.
[
{
"url": "http://metadata.google.internal/computeMetadata/v1/project/project-id",
"body": "",
"statusCode": 502
},
{
"url": "http://metadata/computeMetadata/v1/project/project-id",
"body": "",
"statusCode": 502
},
{
"url": "http://169.254.169.254/computeMetadata/v1/project/project-id",
"body": "",
"statusCode": 502
}
]So lets do just that:
kubectl apply -f istio-egress-rule-metadata.yaml
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
name: metadata-ext
spec:
addresses:
- 169.254.169.254
hosts:
- metadata.google.internal
ports:
- number: 80
name: http
protocol: HTTP
resolution: STATIC
location: MESH_EXTERNAL
endpoints:
- address: 169.254.169.254Try it again and you should see
[
{
"url": "http://metadata.google.internal/computeMetadata/v1/project/project-id",
"statusCode": 200
},
{
"url": "http://metadata/computeMetadata/v1/project/project-id",
"statusCode": 502
},
{
"url": "http://169.254.169.254/computeMetadata/v1/project/project-id",
"statusCode": 200
}
]
Ref: Redefining extensibility in proxies - introducing WebAssembly to Envoy and Istio
The following steps will deploy a trivial wasm module to the cluster that returns hello world back as a header.
We are going to compile and use the main upstream istio envoy example for wasm. The expected output is a new response header and replaced location attribute
see envoy_filter_http_wasm_example.cc
To use this, you need bazel installed and ready to compile a c++ app from scratch
git clone --single-branch --branch release-1.7 https://github.com/istio/envoy.git
bazel build examples/wasm:envoy_filter_http_wasm_example.wasmOnce you have that, upload the binary as a config map called example-filter
kubectl create cm -n default example-filter --from-file=`pwd`/bazel-bin/examples/wasm/envoy_filter_http_wasm_example.wasm Then direct inte sidecar to use a mounted wasm filter
kubectl apply -f fe-v1-wasm-inject.yaml
kubectl replace -f fe-v1-wasm.yamlOnce thats done, invoke the frontend:
$ curl -v --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/version
> GET /version HTTP/2
> Host: 34.123.13.130
> user-agent: curl/7.72.0
> accept: */*
< HTTP/2 200
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 1
< etag: W/"1-NWoZK3kTsExUV00Ywo1G5jlUKKs"
< date: Tue, 22 Sep 2020 18:08:35 GMT
< x-envoy-upstream-service-time: 38
< newheader: newheadervalue
< location: envoy-wasm <<<<<<<<<<<<<<<<<
< server: istio-envoy <<<<<<<<<<<<<<<<<
< You'll see the two new headers.
TODO: use wasme cli
The following will setup a simple Request/Response LUA EnvoyFilter for the frontent myapp:
The settings below injects headers in both the request and response streams:
kubectl apply -f istio-fev1-httpfilter-lua.yamlapiVersion: networking.istio.io/v1
kind: EnvoyFilter
metadata:
name: ui-examplefilter
namespace: default
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
proxy:
proxyVersion: '1\.9.*'
listener:
filterChain:
filter:
name: envoy.http_connection_manager
subFilter:
name: envoy.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.lua
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inlineCode: |
function envoy_on_request(request_handle)
request_handle:headers():add("foo", "bar")
end
function envoy_on_response(response_handle)
response_handle:headers():add("foo2", "bar2")
end
workloadSelector:
labels:
app: myapp
version: v1Note the response headers back to the caller (foo2:bar2) and the echo of the headers as received by the service from envoy (foo:bar)
$ curl -v --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/headerz
> GET /headerz HTTP/2
> Host: 35.184.101.110
> User-Agent: curl/7.60.0
> Accept: */*
< HTTP/2 200
< x-powered-by: Express
< content-type: application/json; charset=utf-8
< contLUAent-length: 626
< etag: W/"272-vkps3sJOT8NW67CxK6gzGw"
< date: Fri, 22 Mar 2019 00:40:36 GMT
< x-envoy-upstream-service-time: 7
< foo2: bar2 <<<<<<<<<
< server: istio-envoy
{
"host": "34.123.13.130",
"user-agent": "curl/7.72.0",
"accept": "*/*",
"x-forwarded-for": "10.128.0.121",
"x-forwarded-proto": "https",
"x-request-id": "52fb996a-2f7a-9352-be5e-025c4328bcae",
"x-envoy-attempt-count": "1",
"content-length": "0",
"x-envoy-internal": "true",
"x-forwarded-client-cert": "By=spiffe://cluster.local/ns/default/sa/myapp-sa;Hash=3cb34c33d650cdc2b9c62ad7d617be39dd778ec34895e03a1a405137a0fcf3f0;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",
"foo": "bar", <<<<<<<<<<<<<<<<
"x-b3-traceid": "ec14fdbbe9ffd9d0499bc785667ed21f",
"x-b3-spanid": "63d7f435af04f296",
"x-b3-parentspanid": "499bc785667ed21f",
"x-b3-sampled": "1"
}You can also see the backend request header by running an echo back of those headers
curl -v --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP https://istio.domain.com/hostz
< HTTP/2 200
< x-powered-by: Express
< content-type: application/json; charset=utf-8
< content-length: 168
< etag: W/"a8-+rQK5xf1qR07k9sBV9qawQ"
< date: Fri, 22 Mar 2019 00:44:30 GMT
< x-envoy-upstream-service-time: 33
< foo2: bar2 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
< server: istio-envoy
The following steps is basically another walkthrough of the RequestAuthentication and AuthorizationPolicy
In this section, we will extend the sample to implement JWT authentication from the client and also use claims within the JWT payload for an enhanced Service Specific Policy.
Specifically, this section will add perimeter Authentication that validates a JWT token at ingress gateway and then RBAC policies at the Service level will further restrict requests.
There are two users: Alice, Bob and two services svc1, svc2. Alice should be allowed to access only svc1, Bob should only access svc2. Both users must present a JWT issued by the same issuer. In this case, a Self Signed JWT certificate issued by Google. You can also use Fireabase/Cloud Identity or any other JWT that provides a JWK URL)
This section involves several steps...first delete any configurations that may still be active. We need to do this because we will create two new services on the frontend svc1, svc2
kubectl delete -f istio-fev1-httpfilter-lua.yaml
kubectl delete -f istio-fev1-httpfilter-ext_authz.yaml
kubectl delete -f istio-fev1-bev1v2.yaml
kubectl delete -f all-istio.yaml
kubectl apply -f istio-lb-certs.yaml
kubectl apply -f istio-ingress-gateway.yaml
kubectl apply -f istio-ingress-ilbgateway.yaml You can verify the configuration that are active by running:
$ kubectl get svc,deployments,po,serviceaccounts,serviceentry,VirtualService,DestinationRule,Secret,GatewaySince the authentication mode described here involes a JWT, we will setup a Google Cloud Service Account the JWT provider. You are ofcourse free to use any identity provide or even Firebase/Cloud Identity
First redeploy an application that has two frontend services svc1, svc2 accessible using the Host: headervalues (svc1.domain.com and svc2.domain.com)
cd auth_rbac_policy/
kubectl apply -f auth-deployment.yaml -f istio-fe-svc1-fe-svc2.yamlCheck the application still works (it should; we didn't apply policies yet yet)
curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP -H "Host: svc1.example.com" -w "\n" https://istio.domain.com/versionApply the authentication policy that checks for a JWT signed by the service account and audience match on the service. THe following policy will allow all three audience values through the ingress gateway but only those JWTs that match the audience for the service through at the service level:
To bootstrap all this, first we need some JWTS. In this case, we will use GCP serice accounts
To bootstrap the sample client, go to the Google Cloud Console and download a service account JSON file as described here. Copy the service account to the auth_rbac_policy/jwt_cli folder and save the JSON file as svc_account.json.
First get the name of the serice account that will sign the JWT:
kubectl apply -f auth-policy.yamlNote that by default we have mTLS and deny by default
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: deny-all-authz-ns
spec:
{}
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default-peerauth
namespace: default
spec:
mtls:
mode: STRICTmake an api call with a malformed authentication header:
$ curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP -H "Host: svc1.example.com" -H "Authorization: Bearer foo" -w "\n" https://istio.domain.com/version
Jwt is not in the form of Header.Payload.Signaturenow try without a header entirely:
$ curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP -H "Host: svc1.example.com" -w "\n" https://istio.domain.com/version
RBAC: access deniedThe error indicates we did not send in the required header. In the next setp, we will use a small sample client library to acquire a JWT. You can also use google OIDC tokens or any other provider (Firebase, Auth0)
The policy above looks for a specific issuer and audience value. THe jwksUri field maps to the public certificate set for a TEST OIDC signer here:
export URL="https://idp-on-cloud-run-3kdezruzua-uc.a.run.app"
export IAT=`date -u +%s`
export EXP=`date -u +%s -d "+3600 seconds"`
envsubst < "alice.tmpl" > "/tmp/alice.json"
envsubst < "bob.tmpl" > "/tmp/bob.json"
envsubst < "bob_no_groups.tmpl" > "/tmp/bob_no_groups.json"
# simply post the JSON Claims...
export TOKEN_ALICE=`curl -s -X POST -d @/tmp/alice.json $URL/token?kid=rsaKeyID_1`
export TOKEN_BOB=`curl -s -X POST -d @/tmp/bob.json $URL/token?kid=rsaKeyID_1`
export TOKEN_BOB_NO_GROUPS=`curl -s -X POST -d @/tmp/bob_no_groups.json $URL/token?kid=rsaKeyID_1`
echo $TOKEN_ALICE
echo $TOKEN_BOB
echo $TOKEN_BOB_NO_GROUPSThe command line utility will generate two tokens with different specifications.
For Alice,
{
"aud": "https://svc1.example.com",
"exp": 1712160374,
"iat": 1712156774,
"iss": "https://idp-on-cloud-run-3kdezruzua-uc.a.run.app",
"sub": "alice"
}And Bob
{
"aud": "https://svc2.example.com",
"exp": 1712160374,
"groups": [
"group1",
"group2"
],
"iat": 1712156774,
"iss": "https://idp-on-cloud-run-3kdezruzua-uc.a.run.app",
"sub": "bob"
}Bob, no groups
{
"aud": "https://svc2.example.com",
"exp": 1712160374,
"iat": 1712156774,
"iss": "https://idp-on-cloud-run-3kdezruzua-uc.a.run.app",
"sub": "bob"
}WARNING the sample code to generate the jwt at the client side uses a service account JWT where the client itself is minting the JWT specifications (meaning it can setup any claimsets it wants, any
subfield.). In reality, you wouild want to use some other mechanism to acquire a token (Auth0, Firebase Custom Claims, etc).
Now inject the token into the Authorization: Bearer header and try to access the protected service:
for i in {1..1000}; do curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP -H "Host: svc1.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "\n" https://istio.domain.com/version; sleep 1; doneThe request should now pass validation and you're in. What we just did is have one policy that globally to the ingress-gateway. Note, we also applied per-service policies in auth-policy.yaml that checks for the aud: value in the inbound token.
What that means is if you use Alice's token to access svc2, you'll see an authentication validation error because that token doesn't have "https://svc2.example.com" in the audience
$ curl --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "\n" https://istio.domain.com/version
Audiences in Jwt are not allowedIn our example, we had a self-signed JWT locally meaning if the end-user had a service account capable of singing, they coudl setup any audience value (i.,e Alice could create a JWT token with the audience of svc). We need to back up and apply addtional controls through RBAC.
The other way is to push the allow/deny decision down from Authentication to Authorization and then using claims on the Authz polic
In auth-policy.yaml, uncomment the AuthorizationPolicy stanza which does the first check for correct audience value in the inbound token for svc2 (the one bob uses):
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: svc2-az
spec:
action: ALLOW
selector:
matchLabels:
app: svc2
rules:
- to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[iss]
values: ["[email protected]"]
- key: request.auth.claims[aud]
values: ["https://svc2.example.com"]
# - key: request.auth.claims[groups]
# values: ["group1", "group2"]
- key: request.auth.claims[sub]
values: ["bob"]kubectl apply -f auth-policy.yamlConsider we have two JWT tokens for Bob:
One with groups
{
"groups": [
"group1",
"group2"
],
"sub": "bob",
"exp": 1571188782,
"iss": "[email protected]",
"iat": 1571185182,
"aud": "https://svc2.example.com"
}And one without
{
"iss": "[email protected]",
"iat": 1571185734,
"sub": "bob",
"exp": 1571189334,
"aud": "https://svc2.example.com"
}Both Tokens allow access through to the serivce because they pass authentication (the audience and subject):
$ curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP \
-H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -o /dev/null -w "%{http_code}\n" https://istio.domain.com/version
200
$ curl -s -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP \
-H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB_NO_GROUPS" -o /dev/null -w "%{http_code}\n" https://istio.domain.com/version
200But what we want to do is deny a request if the token does not include the group header (i know, if Bob had the service account file, he could "just set it"...anyway)
For now, edit auth-policy.yaml and modify the authorization policy for the backend service to make sure the groups are specified and the groups claims are set
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: svc2-az
spec:
selector:
matchLabels:
app: svc2
rules:
- to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[iss]
values: ["[email protected]"]
- key: request.auth.claims[aud]
values: ["https://svc2.example.com"]
- key: request.auth.claims[groups]
values: ["group1", "group2"]
- key: request.auth.claims[sub]
values: ["bob"]Apply again,
kubectl apply -f auth-policy.yamlWait maybe 30seconds (it takes time for the policy to propagte)
Once you set that, only Alice should be able to access svc1 and only Bob access svc2 except when no group info is provided in the JWT
$ curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP \
-H "Host: svc1.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "%{http_code}\n" https://istio.domain.com/version
200
$ curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP \
-H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "%{http_code}\n" https://istio.domain.com/version
403
Audiences in Jwt are not allowed
$ curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP \
-H "Host: svc1.example.com" -H "Authorization: Bearer $TOKEN_BOB" -w "%{http_code}\n" https://istio.domain.com/version
403
Audiences in Jwt are not allowed
$ curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP \
-H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -o /dev/null -w "%{http_code}\n" https://istio.domain.com/version
200
$ curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP \
-H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB_NO_GROUPS" -o /dev/null --w "%{http_code}\n" https://istio.domain.com/version
403
RBAC: access deniedNotice that bob was only allowed in when the token carried group info.
In this section, we extend the working set to allow Alice and Bob to access frontend services and ALSO setup an RBAC policy that allows svcA to access svcB.
When we deployed the application, we associated a service account with each workoad
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: svc1-sa
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: svc2-sa
---We can use this service acount to say: 'only allow requests from svc1-sa to access svc2'. We do this by placing another AuthorizationPolicy policy rule in for svc2
- from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"] That is,
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: svc2-az
spec:
selector:
matchLabels:
app: svc2
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"]
to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[iss]
values: ["[email protected]"]
- key: request.auth.claims[aud]
values: ["https://svc2.example.com"]
- key: request.auth.claims[groups]
values: ["group1", "group2"]
- key: request.auth.claims[sub]
values: ["bob"]Apply again,
kubectl apply -f auth-policy.yaml
Now, if bob tries to access svc2 externally even with a correct token, he will see
$ curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP \
-H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -w "%{http_code}\n" https://istio.domain.com/version
RBAC: access deniedLet try to exc into a pod where svc1 is running and access svc2:
$ kubectl get po
NAME READY STATUS RESTARTS AGE
svc1-77b6bd69cc-bldf8 2/2 Running 0 29m
svc2-bcf6cbd55-hrp25 2/2 Running 0 29m
$ kubectl exec -ti svc1-77b6bd69cc-bldf8 -- /bin/bashFirst try to access the backend service:
curl -s -w "%{http_code}\n" http://svc2.default.svc.cluster.local:8080/version
403
RBAC: access deniedYou'll see a 403 because although the request was inbound from svc1 which is using PEER authentication, we did not add Bob's JWT token. So set an env-var and execute the request again:
root@svc1-7489fbf8d4-8tffm:/# export TOKEN_BOB=eyJhbGciOi...
root@svc1-7489fbf8d4-8tffm:/# curl -s -w "%{http_code}\n" -H "Authorization: Bearer $TOKEN_BOB" http://svc2.default.svc.cluster.local:8080/version
200
You are now in!
This is a bit silly since we needed to use the JWT token for bob for just service to serice traffic.
You dont' ofcourse need to do that: just edit the AuthorizationPolicy for svc2 and comment out
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: svc2-az
spec:
action: ALLOW
selector:
matchLabels:
app: svc2
rules:
- to:
- operation:
methods: ["GET"]
from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"]
# when:
# - key: request.auth.claims[iss]
# values: ["[email protected]"]
# - key: request.auth.claims[aud]
# values: ["https://svc2.example.com"]
# - key: request.auth.claims[groups]
# values: ["group1", "group2"]
# - key: request.auth.claims[sub]
# values: ["bob"]Bob can't access svc2 from the outside but svc1 can access svc2
# apply
$ kubectl apply -f auth-policy.yaml
# from external -> svc1
curl -s --cacert certs/root-ca.crt --resolve istio.domain.com:443:$GATEWAY_IP \
-H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -w "%{http_code}\n" https://istio.domain.com/version
403
RBAC: access denied
# from svc1->svc
curl -s -w "%{http_code}\n" http://svc2.default.svc.cluster.local:8080/version
200You can also setup envoy.ext_authz Filter in this cluster. When using the ext_authz filter on the frontend service, any request for app: myapp, version: v1 will undergo an external authorization check by a serivce you run elsewhere. The external serivice will only allow a request through if it carries Authorizaton: Bearer foo in the header.
apiVersion: networking.istio.io/v1
kind: EnvoyFilter
metadata:
name: ext-authz-filter
spec:
workloadSelector:
labels:
app: myapp
version: v1
configPatches:
- applyTo: HTTP_FILTER
match:
proxy:
proxyVersion: ^1\.7.*
context: SIDECAR_INBOUND
listener:
portNumber: 8080
filterChain:
filter:
name: "envoy.http_connection_manager"
subFilter:
name: "envoy.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
grpc_service:
google_grpc:
target_uri: "your_grpc_server_ip:50051"
stat_prefix: "ext_authz" To use this type of authorization check, you will need to run a serivce somewhere (either within istio (strongly preferred for latency) or external to istio). The following runs the serivce external to istio:
- Istio External Authorization Server
- Envoy External Authorization server (envoy.ext_authz) HelloWorld
First spin up a GCP VM that has an external IP, install golang there and startup the authz server in the git repo provided. You'll also need to open up port :50051 to that VM.
After that, add in the ip address of yoru vm to the yaml file and apply the envoy filter:
kubectl apply -f istio-fev1-httpfilter-ext_authz.yamlOnce you do that, every request to the fronend service will fail unless the specific header is sent through.
Note you'll ofcourse not want to run this serivce anywhere thas externally accessible!...this is just for a demo!!
THis is what an inbound request from istio to the authorization server may look like:
$ go run grpc_server.go
2020/02/20 20:41:40 Starting gRPC Server at :50051
2020/02/20 20:42:41 >>> Authorization called check()
2020/02/20 20:42:41 Inbound Headers:
2020/02/20 20:42:41 {
":authority": "35.238.81.95",
":method": "GET",
":path": "/version",
"accept": "*/*",
"authorization": "Bearer foo",
"content-length": "0",
"user-agent": "curl/7.66.0",
"x-b3-sampled": "0",
"x-b3-spanid": "b228f9e2179794c5",
"x-b3-traceid": "725c448565d59423b228f9e2179794c5",
"x-envoy-internal": "true",
"x-forwarded-client-cert": "By=spiffe://cluster.local/ns/default/sa/myapp-sa;Hash=ae6b57b6ce2932c74b54c40c5e1a7a13daf2828edeb688f9d273a6ea54f38dbf;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",
"x-forwarded-for": "10.128.0.61",
"x-forwarded-proto": "https",
"x-istio-attributes": "CiMKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIHEgVteWFwcAoqCh1kZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWVzcGFjZRIJEgdkZWZhdWx0Ck4KCnNvdXJjZS51aWQSQBI+a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTk3ZGNkN2Y4Ny02MnpjZC5pc3Rpby1zeXN0ZW0KPQoYZGVzdGluYXRpb24uc2VydmljZS5ob3N0EiESH215YXBwLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwKOwoXZGVzdGluYXRpb24uc2VydmljZS51aWQSIBIeaXN0aW86Ly9kZWZhdWx0L3NlcnZpY2VzL215YXBw",
"x-request-id": "e7932678-b3a2-40b8-bc49-6b645448ae28"
}The easiest way to clean up what you did here is to delete the GKE cluster!
gcloud container clusters delete cluster-1
The steps i outlined above is just a small set of what Istio has in store. I'll keep updating this as it move towards 1.0 and subsequent releases.