Skip to content

Commit

Permalink
installation: kubernetes: Document Windows deployment (fluent#353)
Browse files Browse the repository at this point in the history
This explains the basic concept of Kubernetes deployment (e.g. which
file contain what), and shows how to set up Fluent Bit for Windows
logging.

It will be useful for Windows users to get started, and should resolve
the issue fluent-bit#2087.

Signed-off-by: Fujimoto Seiji <[email protected]>
  • Loading branch information
fujimotos authored Oct 9, 2020
1 parent 5990d3d commit f47f097
Showing 1 changed file with 107 additions and 0 deletions.
107 changes: 107 additions & 0 deletions installation/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,3 +92,110 @@ The default configuration of Fluent Bit makes sure of the following:
* The default backend in the configuration is Elasticsearch set by the [Elasticsearch Ouput Plugin](https://docs.fluentbit.io/manual/v/1.0/output/elasticsearch). It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.
* There is an option called **Retry\_Limit** set to False, that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeed.

## Windows Deployment

Since v1.5.0, Fluent Bit supports deployment to Windows pods.

### Log files overview

When deploying Fluent Bit to Kubernetes, there are three log files that you need to pay attention to.

`C:\k\kubelet.err.log`

* This is the error log file from kubelet daemon running on host.
* You will need to retain this file for future troubleshooting (to debug deployment failures etc.)

`C:\var\log\containers\<pod>_<namespace>_<container>-<docker>.log`

* This is the main log file you need to watch. Configure Fluent Bit to follow this file.
* It is actually a symlink to the Docker log file in `C:\ProgramData\`, with some additional metadata on its file name.

`C:\ProgramData\Docker\containers\<docker>\<docker>.log`

* This is the log file produced by Docker.
* Normally you don't directly read from this file, but you need to make sure that this file is visible from Fluent Bit.

Typically, your deployment yaml contains the following volume configuration.

```yaml
spec:
containers:
- name: fluent-bit
image: my-repo/fluent-bit:1.5.0
volumeMounts:
- mountPath: C:\k
name: k
- mountPath: C:\var\log
name: varlog
- mountPath: C:\ProgramData
name: progdata
volumes:
- name: k
hostPath:
path: C:\k
- name: varlog
hostPath:
path: C:\var\log
- name: progdata
hostPath:
path: C:\ProgramData
```
### Configure Fluent Bit
Assuming the basic volume configuration described above, you can apply the following config to start logging.
```yaml
fluent-bit.conf: |
[SERVICE]
Parsers_File C:\\fluent-bit\\parsers.conf
[INPUT]
Name tail
Tag kube.*
Path C:\\var\\log\\containers\\*.log
Parser docker
DB C:\\fluent-bit\\tail_docker.db
Mem_Buf_Limit 7MB
Refresh_Interval 10
[INPUT]
Name tail
Tag kubelet.err
Path C:\\k\\kubelet.err.log
DB C:\\fluent-bit\\tail_kubelet.db
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
[OUTPUT]
Name stdout
Match *
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
```
### Mitigate unstable network on Windows pods
Windows pods often lack working DNS immediately after boot ([#78479](https://github.com/kubernetes/kubernetes/issues/78479)). To mitigate this issue, `filter_kubernetes` provides a built-in mechanism to wait until the network starts up:

* `DNS_Retries` - Retries N times until the network start working (6)
* `DNS_Wait_Time` - Lookup interval between network status checks (30)

By default, Fluent Bit waits for 3 minutes (30 seconds x 6 times). If it's not enough for you, tweak the configuration as follows.

```
[filter]
Name kubernetes
...
DNS_Retries 10
DNS_Wait_Time 30
```

0 comments on commit f47f097

Please sign in to comment.