Skip to content

Commit

Permalink
Added a new directory for continuous pipeline configuration when make…
Browse files Browse the repository at this point in the history
… dist (#1926)
  • Loading branch information
quzard authored Dec 3, 2024
1 parent 0254c5c commit dbe06c7
Show file tree
Hide file tree
Showing 6 changed files with 67 additions and 66 deletions.
80 changes: 40 additions & 40 deletions docs/cn/installation/start-with-k8s.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,39 +8,39 @@

## 采集Kubernetes容器日志

1. 创建部署iLogtail的命名空间
1. 创建部署loongcollector的命名空间

将下面内容保存为ilogtail-ns.yaml
将下面内容保存为loongcollector-ns.yaml

```yaml {.line-numbers}
apiVersion: v1
kind: Namespace
metadata:
name: ilogtail
name: loongcollector
```
您也可以直接从下面的地址下载示例配置。
```bash
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-ns.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/loongcollector-ns.yaml
```

应用上述配置

```bash
kubectl apply -f ilogtail-ns.yaml
kubectl apply -f loongcollector-ns.yaml
```

2. 创建配置iLogtail的ConfigMap和Secret
2. 创建配置loongcollector的ConfigMap和Secret

将下面内容保存为ilogtail-user-configmap.yaml。该ConfigMap后续将作为配置目录挂载到iLogtail容器中,因此可包含多个采集配置。
将下面内容保存为loongcollector-user-configmap.yaml。该ConfigMap后续将作为配置目录挂载到loongcollector容器中,因此可包含多个采集配置。

```yaml {.line-numbers}
apiVersion: v1
kind: ConfigMap
metadata:
name: ilogtail-user-cm
namespace: ilogtail
name: loongcollector-user-cm
namespace: loongcollector
data:
nginx_stdout.yaml: |
enable: true
Expand Down Expand Up @@ -71,14 +71,14 @@
OnlyStdout: true
```

将下面内容保存为ilogtail-secret.yaml。该Secret为可选,当需要将日志写入SLS时会用到。
将下面内容保存为loongcollector-secret.yaml。该Secret为可选,当需要将日志写入SLS时会用到。

```yaml {.line-numbers}
apiVersion: v1
kind: Secret
metadata:
name: ilogtail-secret
namespace: ilogtail
name: loongcollector-secret
namespace: loongcollector
type: Opaque
data:
access_key_id: # base64 accesskey id if you want to flush to SLS
Expand All @@ -88,42 +88,42 @@
您也可以直接从下面的地址下载示例配置。

```bash
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-user-configmap.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-secret.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/loongcollector-user-configmap.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/loongcollector-secret.yaml
```

应用上述配置

```bash
kubectl apply -f ilogtail-user-configmap.yaml
kubectl apply -f ilogtail-secret.yaml
kubectl apply -f loongcollector-user-configmap.yaml
kubectl apply -f loongcollector-secret.yaml
```

3. 创建iLogtail DaemonSet
3. 创建loongcollector DaemonSet

将下面内容保存为ilogtail-daemonset.yaml。
将下面内容保存为loongcollector-daemonset.yaml。

```yaml {.line-numbers}
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ilogtail-ds
namespace: ilogtail
name: loongcollector-ds
namespace: loongcollector
labels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
spec:
selector:
matchLabels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
template:
metadata:
labels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
spec:
tolerations:
- operator: Exists # deploy on all nodes
containers:
- name: logtail
- name: loongcollector
env:
- name: ALIYUN_LOG_ENV_TAGS # add log tags from env
value: _node_name_|_node_ip_
Expand All @@ -137,24 +137,24 @@
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: cpu_usage_limit # iLogtail's self monitor cpu limit
- name: cpu_usage_limit # loongcollector's self monitor cpu limit
value: "1"
- name: mem_usage_limit # iLogtail's self monitor mem limit
- name: mem_usage_limit # loongcollector's self monitor mem limit
value: "512"
- name: default_access_key_id # accesskey id if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
name: loongcollector-secret
key: access_key_id
optional: true
- name: default_access_key # accesskey secret if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
name: loongcollector-secret
key: access_key
optional: true
image: >-
sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:latest
sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/loongcollector-community-edition/loongcollector:latest
imagePullPolicy: IfNotPresent
resources:
limits:
Expand All @@ -170,9 +170,9 @@
mountPropagation: HostToContainer
name: root
readOnly: true
- mountPath: /usr/local/ilogtail/checkpoint # for checkpoint between container restart
- mountPath: /usr/local/loongcollector/data # for checkpoint between container restart
name: checkpoint
- mountPath: /usr/local/ilogtail/config/local # mount config dir
- mountPath: /usr/local/loongcollector/conf/continuous_pipeline_config/local # mount config dir
name: user-config
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
Expand All @@ -187,27 +187,27 @@
type: Directory
name: root
- hostPath:
path: /etc/ilogtail-ilogtail-ds/checkpoint
path: /etc/loongcollector-loongcollector-ds/checkpoint
type: DirectoryOrCreate
name: checkpoint
- configMap:
defaultMode: 420
name: ilogtail-user-cm
name: loongcollector-user-cm
name: user-config
```

您也可以直接从下面的地址下载示例配置。

```bash
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-daemonset.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/loongcollector-daemonset.yaml
```

添加参数示例

```yaml
- name: logtail
- name: loongcollector
command:
- /usr/local/ilogtail/ilogtail_control.sh
- /usr/local/loongcollector/loongcollector_control.sh
args:
- "start_and_block"
- "-enable_containerd_upper_dir_detect=true"
Expand All @@ -218,7 +218,7 @@
应用上述配置

```bash
kubectl apply -f ilogtail-daemonset.yaml
kubectl apply -f loongcollector-daemonset.yaml
```

4. 部署用来测试的nginx
Expand Down Expand Up @@ -271,19 +271,19 @@
5. 发送请求构造示例日志

```bash
kubectl exec nginx-<pod-id> -- curl localhost/hello/ilogtail
kubectl exec nginx-<pod-id> -- curl localhost/hello/loongcollector
```

6. 查看采集到的测试容器标准输出日志

```bash
kubectl logs ilogtail-ds-<pod-id> -n ilogtail
kubectl logs loongcollector-ds-<pod-id> -n loongcollector
```

结果为

```json
2022-07-14 16:36:50 {"_time_":"2022-07-15T00:36:48.489153485+08:00","_source_":"stdout","_image_name_":"docker.io/library/nginx:latest","_container_name_":"nginx","_pod_name_":"nginx-76d49876c7-r892w","_namespace_":"default","_pod_uid_":"07f75a79-da69-40ac-ae2b-77a632929cc6","_container_ip_":"10.223.0.154","remote_addr":"::1","remote_user":"-","time_local":"14/Jul/2022:16:36:48","method":"GET","url":"/hello/ilogtail","protocol":"HTTP/1.1","status":"404","body_bytes_sent":"153","http_referer":"-","http_user_agent":"curl/7.74.0","http_x_forwarded_for":"-","__time__":"1657816609"}
2022-07-14 16:36:50 {"_time_":"2022-07-15T00:36:48.489153485+08:00","_source_":"stdout","_image_name_":"docker.io/library/nginx:latest","_container_name_":"nginx","_pod_name_":"nginx-76d49876c7-r892w","_namespace_":"default","_pod_uid_":"07f75a79-da69-40ac-ae2b-77a632929cc6","_container_ip_":"10.223.0.154","remote_addr":"::1","remote_user":"-","time_local":"14/Jul/2022:16:36:48","method":"GET","url":"/hello/loongcollector","protocol":"HTTP/1.1","status":"404","body_bytes_sent":"153","http_referer":"-","http_user_agent":"curl/7.74.0","http_x_forwarded_for":"-","__time__":"1657816609"}
```

## 采集模版
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2022 iLogtail Authors
# Copyright 2022 loongcollector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,23 +15,23 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ilogtail-ds
namespace: ilogtail
name: loongcollector-ds
namespace: loongcollector
labels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
spec:
selector:
matchLabels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
template:
metadata:
labels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
spec:
tolerations:
- operator: Exists # deploy on all nodes
containers:
- name: logtail
- name: loongcollector
env:
- name: ALIYUN_LOG_ENV_TAGS # add log tags from env
value: _node_name_|_node_ip_
Expand All @@ -45,24 +45,24 @@ spec:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: cpu_usage_limit # iLogtail's self monitor cpu limit
- name: cpu_usage_limit # loongcollector's self monitor cpu limit
value: "1"
- name: mem_usage_limit # iLogtail's self monitor mem limit
- name: mem_usage_limit # loongcollector's self monitor mem limit
value: "512"
- name: default_access_key_id # accesskey id if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
name: loongcollector-secret
key: access_key_id
optional: true
- name: default_access_key # accesskey secret if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
name: loongcollector-secret
key: access_key
optional: true
image: >-
sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:latest
sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/loongcollector-community-edition/loongcollector:latest
imagePullPolicy: IfNotPresent
resources:
limits:
Expand All @@ -78,16 +78,16 @@ spec:
mountPropagation: HostToContainer
name: root
readOnly: true
- mountPath: /usr/local/ilogtail/checkpoint # for checkpoint between container restart
- mountPath: /usr/local/loongcollector/data # for checkpoint between container restart
name: checkpoint
- mountPath: /usr/local/ilogtail/config/local # mount config dir
- mountPath: /usr/local/loongcollector/conf/continuous_pipeline_config/local # mount config dir
name: user-config
readOnly: true
lifecycle:
preStop:
exec:
command:
- /usr/local/ilogtail/ilogtail_control.sh
- /usr/local/loongcollector/loongcollector_control.sh
- stop
- "3"
livenessProbe:
Expand All @@ -112,10 +112,10 @@ spec:
type: Directory
name: root
- hostPath:
path: /etc/ilogtail-ilogtail-ds/checkpoint
path: /etc/loongcollector-loongcollector-ds/checkpoint
type: DirectoryOrCreate
name: checkpoint
- configMap:
defaultMode: 420
name: ilogtail-user-cm
name: loongcollector-user-cm
name: user-config
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2022 iLogtail Authors
# Copyright 2022 loongcollector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,4 +15,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: ilogtail
name: loongcollector
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2022 iLogtail Authors
# Copyright 2022 loongcollector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,8 +15,8 @@
apiVersion: v1
kind: Secret
metadata:
name: ilogtail-secret
namespace: ilogtail
name: loongcollector-secret
namespace: loongcollector
type: Opaque
data:
access_key_id: # accesskey id if you want to flush to SLS
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2022 iLogtail Authors
# Copyright 2022 loongcollector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,8 +15,8 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: ilogtail-user-cm
namespace: ilogtail
name: loongcollector-user-cm
namespace: loongcollector
data:
nginx_stdout.yaml: |
enable: true
Expand Down
3 changes: 2 additions & 1 deletion scripts/dist.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,9 @@ cp "${ROOTDIR}/${OUT_DIR}/loongcollector" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}
cp "${ROOTDIR}/${OUT_DIR}/libGoPluginAdapter.so" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}"
cp "${ROOTDIR}/${OUT_DIR}/libGoPluginBase.so" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}"
mkdir -p "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf/instance_config/local/"
mkdir -p "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf/continuous_pipeline_config/local/"
cp "${ROOTDIR}/${OUT_DIR}/conf/instance_config/local/loongcollector_config.json" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf/instance_config/local/"
cp -a "${ROOTDIR}/${OUT_DIR}/conf/continuous_pipeline_config/local" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf"
cp -a "${ROOTDIR}/${OUT_DIR}/conf/continuous_pipeline_config/local" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf/continuous_pipeline_config"
if file "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/loongcollector" | grep x86-64; then ./scripts/download_ebpflib.sh "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}"; fi

# Splitting debug info at build time with -gsplit-dwarf does not work with current gcc version
Expand Down

0 comments on commit dbe06c7

Please sign in to comment.