Skip to content

Latest commit

 

History

History
183 lines (135 loc) · 7.98 KB

logging-monitoring.adoc

File metadata and controls

183 lines (135 loc) · 7.98 KB

Container Adoption Lab

Logging and Monitoring

Expected Outcome:

  • Explore monitoring and logging solutions

Lab Requirements:

  • Kubernetes cluster

Average Lab Time: 30 - 45 minutes

Introduction

The Pet Store application is up and running in our Kubernetes cluster. We need to provide visibility on the performance and activity of the application running in various environments. In this lab we will discover different methods for gathering and exposing these metrics.

Getting started

  1. To get started, switch to the Lab6 folder within this repository

Monitoring with cAdvisor and dashboard

Open up port 4194 on security groups for master/node instances and then navigate to:

This will display the cAdvisor dashboard which shows cpu, memory and file system metrics for this particular node.

Deploy the kubernetes dashboard:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

Dashboard can be seen using the following command:

kubectl proxy --address 0.0.0.0 --accept-hosts '.*' --port 8080

Now, Dashboard is accessible via Preview, Preview Running Application as:

https://ENVIRONMENT_ID.vfs.cloud9.REGION_ID.amazonaws.com/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

Where ENVIRONMENT_ID is your Cloud9 IDE environment id (you should see it once click the built-in browser address bar) and REGION_ID is AWS region id (e.g. us-west-2).

Starting with Kubernetes 1.7, Dashboard supports authentication. Read more about it at https://github.com/kubernetes/dashboard/wiki/Access-control#introduction. We’ll use a bearer token for authentication.

Check existing secrets in the kube-system namespace:

$ kubectl -n kube-system get secret
NAME                                     TYPE                                  DATA      AGE
attachdetach-controller-token-dhkcr      kubernetes.io/service-account-token   3         3h
certificate-controller-token-p131b       kubernetes.io/service-account-token   3         3h
daemon-set-controller-token-r4mmp        kubernetes.io/service-account-token   3         3h
default-token-7vh0x                      kubernetes.io/service-account-token   3         3h
deployment-controller-token-jlzkj        kubernetes.io/service-account-token   3         3h
disruption-controller-token-qrx2v        kubernetes.io/service-account-token   3         3h
dns-controller-token-v49b6               kubernetes.io/service-account-token   3         3h
endpoint-controller-token-hgkbm          kubernetes.io/service-account-token   3         3h
generic-garbage-collector-token-34fvc    kubernetes.io/service-account-token   3         3h
horizontal-pod-autoscaler-token-lhbkf    kubernetes.io/service-account-token   3         3h
job-controller-token-c2s8j               kubernetes.io/service-account-token   3         3h
kube-dns-autoscaler-token-s3svx          kubernetes.io/service-account-token   3         3h
kube-dns-token-92xzb                     kubernetes.io/service-account-token   3         3h
kube-proxy-token-0ww14                   kubernetes.io/service-account-token   3         3h
kubernetes-dashboard-certs               Opaque                                2         9m
kubernetes-dashboard-key-holder          Opaque                                2         9m
kubernetes-dashboard-token-vt0fd         kubernetes.io/service-account-token   3         10m
namespace-controller-token-423gh         kubernetes.io/service-account-token   3         3h
node-controller-token-r6lsr              kubernetes.io/service-account-token   3         3h
persistent-volume-binder-token-xv30g     kubernetes.io/service-account-token   3         3h
pod-garbage-collector-token-fwmv4        kubernetes.io/service-account-token   3         3h
replicaset-controller-token-0cg8r        kubernetes.io/service-account-token   3         3h
replication-controller-token-3fwxd       kubernetes.io/service-account-token   3         3h
resourcequota-controller-token-6rl9f     kubernetes.io/service-account-token   3         3h
route-controller-token-9brzb             kubernetes.io/service-account-token   3         3h
service-account-controller-token-bqlsk   kubernetes.io/service-account-token   3         3h
service-controller-token-1qlg6           kubernetes.io/service-account-token   3         3h
statefulset-controller-token-kmgzg       kubernetes.io/service-account-token   3         3h
ttl-controller-token-vbnhf               kubernetes.io/service-account-token   3         3h

We can login using any secret with type 'kubernetes.io/service-account-token', though each of them have different privileges. In our case, we’ll use the token from secret default-token-7vh0x to login. Use the following command to get the token for this secret:

kubectl -n kube-system describe secret default-token-7vh0x

Note you’ll need to replace default-token-7vh0x with the default-token from your output list.

It shows the output:

Name:         default-token-7vh0x
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name=default
              kubernetes.io/service-account.uid=3a3fea86-b3a1-11e7-9d90-06b1e747c654

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1046 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLTd2aDB4Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzYTNmZWE4Ni1iM2ExLTExZTctOWQ5MC0wNmIxZTc0N2M2NTQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.GHW-7rJcxmvujkClrN6heOi_RYlRivzwb4ScZZgGyaCR9tu2V0Z8PE5UR6E_3Vi9iBCjuO6L6MLP641bKoHB635T0BZymJpSeMPQ7t1F02BsnXAbyDFfal9NUSV7HoPAhlgURZWQrnWojNlVIFLqhAPO-5T493SYT56OwNPBhApWwSBBGdeF8EvAHGtDFBW1EMRWRt25dSffeyaBBes5PoJ4SPq4BprSCLXPdt-StPIB-FyMx1M-zarfqkKf7EJKetL478uWRGyGNNhSfRC-1p6qrRpbgCdf3geCLzDtbDT2SBmLv1KRjwMbW3EF4jlmkM4ZWyacKIUljEnG0oltjA

Copy the value of token from this output, select Token in the Dashboard login window, and paste the text. Click on SIGN IN to see the default Dashboard view:

kubernetes dashboard default

Click on Nodes to see a textual representation about the nodes running in the cluster:

monitoring nodes before

From the dashboard you will be able to see metrics that are rolled up from the entire cluster in one place. Browse around the explore the dashboard. Although most kubernetes users prefer to interact with a particular cluster, the dashboard offers a GUI from where you can manage your applications and infrastructure.

Most users will deploy additional monitoring and logging tools, like Prometheus which is out of scope for this workshop but something you should explore for more detailed monitoring capabilities.

Logging

For basic ad-hoc logging you can use kubernetes built in tooling for logging:

kubectl get pods

Then copy one of the pod names and:

kubectl logs <pod name>

Anything that the container/pod streams to STDOUT will be displayed.

For cluster level logging many options exist, one of them is to deploy an in-cluster Elasticsearch and Kibana environment, that can be done with:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/logging-elasticsearch/v1.6.0.yaml

Then:

kubectl proxy --address 0.0.0.0 --accept-hosts '.*' --port 8080

Then you can see the url for the kibana dashboard:

kubectl cluster-info

Open the kibana URL, the username is admin and password you retrieved from:

kubectl config view