init iLogtail

CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), but make some changes.
+
+## Release tags
+- `public` for changes that will be published in release notes.
+- `inner` for other changes.
+
+
+## Platform tags
+This project will generate binaries for at least two platforms (Linux/Windows.noarch), so please add platform tag for your changes, such as:
+
+- `both` for all platforms.
+- `win` for Windows.
+- `linux` for Linux.
+- `linux.arm` for Linux on ARM.
+- ....
+
+
+## Type of changes
+- `added` for new features.
+- `updated` for changes in existing functionality.
+- `deprecated` for soon-to-be removed features.
+- `removed` for now removed features.
+- `fixed` for any bug fixes.
+- `security` in case of vulnerabilities.
+
+
+## [Unreleased]

CODE_OF_CONDUCT.md

@@ -0,0 +1,5 @@
+# Code of Conduct
+
+Alibaba has adopted a Code of Conduct that we expect project participants to adhere to.
+
+Please refer to [Alibaba Open Source Code of Conduct](https://github.com/AlibabaDR/community/blob/master/CODE_OF_CONDUCT.md) ([中文版](https://github.com/AlibabaDR/community/blob/master/CODE_OF_CONDUCT_zh.md)).

LICENSE

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

Makefile

@@ -0,0 +1,142 @@
+# Copyright 2021 iLogtail Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+VERSION ?= latest
+DOCKER_TYPE ?= default
+SCOPE ?= .
+BINARY = logtail-plugin
+
+TEST_DEBUG ?= false
+TEST_PROFILE ?= false
+TEST_SCOPE ?= "all"
+
+PLATFORMS := linux darwin windows
+
+OS = $(shell uname)
+ARCH = amd64
+
+GO = go
+GO_PATH = $$($(GO) env GOPATH)
+GO_BUILD = $(GO) build
+GO_GET = $(GO) get
+GO_TEST = $(GO) test
+GO_LINT = golangci-lint
+GO_ADDLICENSE = $(GO_PATH)/bin/addlicense
+GO_PACKR = $(GO_PATH)/bin/packr2
+GO_BUILD_FLAGS = -v
+
+LICENSE_COVERAGE_FILE=license_coverage.txt
+OUT_DIR = bin
+
+.PHONY: tools
+tools:
+	if [ $(OS) == Darwin ]; then \
+	    $(GO_LINT) version || brew install golangci-lint ; \
+	else \
+  		$(GO_LINT) version || curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(GO_PATH)/bin v1.39.0; \
+  	fi
+	$(GO_ADDLICENSE) version || GO111MODULE=off $(GO_GET) -u github.com/google/addlicense
+
+
+.PHONY: clean
+clean:
+	rm -rf $(LICENSE_COVERAGE_FILE)
+	rm -rf $(OUT_DIR)
+	rm -rf behavior-test
+	rm -rf performance-test
+	rm -rf core-test
+	rm -rf e2e-engine-coverage.txt
+	rm -rf find_licenses
+
+.PHONY: license
+license:  tools
+	./scripts/package_license.sh add-license $(SCOPE)
+
+.PHONY: check-license
+check-license: clean
+	./scripts/package_license.sh check $(SCOPE) > $(LICENSE_COVERAGE_FILE)
+
+.PHONY: lint
+lint: tools
+	$(GO_LINT) run -v --timeout 5m $(SCOPE)/... && make lint-pkg && make lint-e2e
+
+.PHONY: lint-pkg
+lint-pkg: tools
+	cd pkg && pwd && $(GO_LINT) run -v --timeout 5m ./...
+
+.PHONY: lint-test
+lint-e2e: tools
+	cd test && pwd && $(GO_LINT) run -v --timeout 5m ./...
+
+.PHONY: build
+build: clean
+	./scripts/build.sh vendor default
+
+.PHONY: cgobuild
+cgobuild: clean
+	./scripts/build.sh vendor c-shared
+
+.PHONY: gocbuild
+gocbuild: clean
+	./scripts/gocbuild.sh
+
+.PHONY: docker
+docker:
+	./scripts/docker-build.sh $(VERSION) $(DOCKER_TYPE)
+
+# coveragedocker compile with goc to analysis the coverage in e2e testing
+coveragedocker:
+	./scripts/docker-build.sh $(VERSION) coverage
+
+# provide a goc server for e2e testing
+.PHONY: gocdocker
+gocdocker:
+	docker build -t goc-server:latest  --no-cache . -f ./docker/Dockerfile_goc
+
+.PHONY: vendor
+vendor:
+	rm -rf vendor
+	$(GO) mod vendor
+	./external/sync_vendor.py
+
+.PHONY: check-dependency-licenses
+check-dependency-licenses: clean
+	./scripts/dependency_licenses.sh main LICENSE_OF_ILOGTAIL_DEPENDENCIES.md && ./scripts/dependency_licenses.sh test LICENSE_OF_TESTENGINE_DEPENDENCIES.md
+
+.PHONY: e2e-docs
+e2e-docs: clean
+	cd test && go build -o logtailplugin-test-tool  . && ./logtailplugin-test-tool docs && rm -f logtailplugin-test-tool
+
+# e2e test
+.PHONY: e2e
+e2e: clean gocdocker coveragedocker
+	TEST_DEBUG=$(TEST_DEBUG) TEST_PROFILE=$(TEST_PROFILE)  ./scripts/e2e.sh behavior $(TEST_SCOPE)
+
+.PHONY: e2e-core
+e2e-core: clean gocdocker coveragedocker
+	TEST_DEBUG=$(TEST_DEBUG) TEST_PROFILE=$(TEST_PROFILE)  ./scripts/e2e.sh core $(TEST_SCOPE)
+
+.PHONY: e2e-performance
+e2e-performance: clean docker
+	TEST_DEBUG=$(TEST_DEBUG) TEST_PROFILE=$(TEST_PROFILE)  ./scripts/e2e.sh performance $(TEST_SCOPE)
+
+# unit test
+.PHONY: test-e2e-engine
+test-e2e-engine: clean
+	cd test && go test  ./... -coverprofile=../e2e-engine-coverage.txt -covermode=atomic -tags docker_ready
+
+.PHONY: test
+test: clean
+	cp pkg/logtail/libPluginAdapter.so ./main
+	go test $$(go list ./...|grep -v vendor |grep -v telegraf|grep -v external|grep -v envconfig) -coverprofile .testCoverage.txt

README-cn.md

@@ -0,0 +1,87 @@
+Alibaba iLogtail - 阿里轻量级遥测数据采集端 | [English](./README.md)
+==========
+<img src="https://sls-opensource.oss-us-west-1.aliyuncs.com/ilogtail/ilogtail.svg?versionId=CAEQMxiBgIDEmq.m6BciIDkzNmE2OWU4NzIwZjQ1Y2ZiYmIxZjhiYjMyNmQxZTdi" alt="ilogtail logo" height="150px" align="right" />
+
+iLogtail 为可观测场景而生，拥有的轻量级、高性能、自动化配置等诸多生产级别特性，在阿里巴巴以及外部数万家阿里云客户内部广泛应用。你可以将它部署于物理机，虚拟机，Kubernetes等多种环境中来采集遥测数据，例如logs、traces和metrics。
+
+[![GitHub stars](https://img.shields.io/github/stars/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/stargazers)
+[![GitHub issues](https://img.shields.io/github/issues/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/issues)
+[![GitHub license](https://img.shields.io/github/license/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/blob/main/LICENSE)
+
+**iLogtail** 的核心优势主要有：
+* 支持多种Logs、Traces、Metrics数据采集，尤其对容器、Kubernetes环境支持非常友好
+* 数据采集资源消耗极低，相比同类遥测数据采集的Agent性能好5-20倍
+* 高稳定性，在阿里巴巴以及数万阿里云客户生产中使用验证，部署量近千万，每天采集数十PB可观测数据
+* 支持插件化扩展，可任意扩充数据采集、处理、聚合、发送模块
+* 支持配置远程管理，支持以图形化、SDK、K8s Operator等方式进行配置管理，可轻松管理百万台机器的数据采集
+* 支持自监控、流量控制、资源控制、主动告警、采集统计等多种高级特性
+
+**iLogtail** 支持收集多种遥测数据并将其传输到多种不同的后端，例如 [SLS可观测平台](https://help.aliyun.com/product/28958.html) 。 支持采集的数据主要如下:
+- Logs
+    - 收集静态日志文件
+    - 在容器化环境中运行时动态收集文件
+    - 在容器化环境中运行时动态收集 Stdout
+- Traces
+    - OpenTelemetry 协议
+    - Skywalking V2 协议
+    - Skywalking V3 协议
+    - ...
+- Metrics
+    - Node指标
+    - Process指标
+    - GPU 指标
+    - Nginx 指标
+    - 支持获取Prometheus指标
+    - 支持收集Telegraf指标
+    - ...
+
+
+
+
+# 快速开始
+
+本仓库是**iLogtail**的golang部分，包含了iLogtail的大部分功能。 而且，它可以单独工作，也可以使用 CGO 与 iLogtail-C（即将开源）一起工作。
+
+1.本地启动
+
+```shell
+make build
+./bin/logtailplugin
+```
+
+2、阿里云启动
+
+请阅读此 [doc](https://help.aliyun.com/document_detail/65018.html)。
+
+# 文档
+
+有关最新版本的文档，请参阅 [文档索引](./docs/zh/README.md)
+
+- [Input 插件](./docs/zh/guides/How-to-write-input-plugins.md)
+- [Processor 插件](./docs/zh/guides/How-to-write-processor-plugins.md)
+- [Aggregator 插件](./docs/zh/guides/How-to-write-aggregator-plugins.md)
+- [Flusher 插件](./docs/zh/guides/How-to-write-flusher-plugins.md)
+
+# 贡献
+
+- [修复和报告错误](https://github.com/alibaba/ilogtail/issues)
+- [改进文档](https://github.com/alibaba/ilogtail/labels/documentation)
+- [审查代码和功能提案](https://github.com/alibaba/ilogtail/pulls)
+- [贡献插件](./docs/zh/guides/README.md)
+
+# 联系我们
+您可以通过[Github Issues](https://github.com/alibaba/ilogtail/issues) 报告bug、提出建议或参与讨论，或通过以下方式联系我们：
+
+- 钉钉：33867413
+- 微信：日志服务
+- 哔哩哔哩：[阿里云SLS](https://space.bilibili.com/630680534?from=search&seid=2845737427240690794&spm_id_from=333.337.0.0)
+- 知乎：[阿里云日志服务](https://www.zhihu.com/people/a-li-yun-ri-zhi-fu-wu)
+
+<img src="https://sls-opensource.oss-us-west-1.aliyuncs.com/ilogtail/ilogtail-contact.png?versionId=CAEQMxiBgIC.jPui6RciIDhlNzVjYmQ5OTIwNzQ4NTBiMzk1YTgyN2VkNmU5MDBi" alt="ilogtail logo" style="width: 100%; height: 100%" />
+
+# 我们的用户
+
+数以万计的公司在阿里云、线下IDC、其他云等多种环境中使用 iLogtail。 更多详情请看[这里](https://help.aliyun.com/document_detail/250269.html) 。
+
+# Licence
+[Apache 2.0 许可证](./LICENSE)

README.md

@@ -1 +1,86 @@
-# ilogtail
+Alibaba iLogtail - The Lightweight Collector of SLS in Alibaba Cloud | [中文版本](README-cn.md)
+==========
+<img src="https://sls-opensource.oss-us-west-1.aliyuncs.com/ilogtail/ilogtail.svg?versionId=CAEQMxiBgIDEmq.m6BciIDkzNmE2OWU4NzIwZjQ1Y2ZiYmIxZjhiYjMyNmQxZTdi" alt="ilogtail logo" height="150px" align="right" />
+
+iLogtail was born for observable scenarios and has many production-level features such as lightweight, high performance, and automated configuration, which are widely used internally by Alibaba Group and tens of thousands of external Alibaba Cloud customers. You can deploy it in physical machines, Kubernetes and other environments to collect telemetry data, such as logs, traces and metrics.
+
+[![GitHub stars](https://img.shields.io/github/stars/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/stargazers)
+[![GitHub issues](https://img.shields.io/github/issues/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/issues)
+[![GitHub license](https://img.shields.io/github/license/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/blob/main/LICENSE)
+
+# Abstract
+
+The core advantages of **iLogtail** :
+* Support a variety of Logs, Traces, Metrics data collection, and friendly to container and Kubernetes environment support.
+* The resource cost of data collection is quite low, 5-20 times better than similar telemetry data collection Agent performance.
+* High stability, used in the production of Alibaba and tens of thousands of Alibaba Cloud customers,  and collecting dozens of petabytes of observable data every day with nearly tens of millions deployments.
+* Support plugin expansion, such as collection, processing, aggregation, and sending modules.
+* Support configuration remote management and provide a variety of ways, such as SLS console, SDK, K8s Operator, etc.
+* Supports multiple advanced features such as self-monitoring, flow control, resource control, alarms, and statistics collection.
+
+
+**iLogtail** supports the collection of a variety of telemetry data and transmission to a variety of different backends, such as [SLS observable platform](https://www.aliyun.com/product/sls?spm=5176.19720258.J_8058803260.41.68232c4a8fRbNM). The data supported for collection are mainly as follows:
+- Logs
+  - Collect static log files
+  - Dynamic collect the files when running with containerized environment
+  - Dynamic collect Stdout when running with containerized environment
+- Traces
+  - OpenTelemetry protocol
+  - Skywalking V2 protocol
+  - Skywalking V3 protocol
+  - ...
+Metrics
+  - Node metrics
+  - Process metrics
+  - Gpu metrics
+  - Nginx metrics
+  - Support fetch prometheus metrics
+  - Support transfer telegraf metrics
+  - ...
+
+# Quick Start
+This repository is the golang part of **iLogtail**，it contains most of the features of iLogtail. And, it can work by itself or work with Ilogtail-C(open source soon) using CGO.
+
+1. Start with local
+
+```shell
+make build && sh bin/logtailplugin
+```
+2. Start with Alibaba Cloud  
+Please read this [doc](https://www.alibabacloud.com/help/doc-detail/28979.htm?spm=a2c63.l28256.b99.76.79c84211P7kJzN).
+
+# Documentation
+
+For documentation on the latest version see the [documentation index](./docs/en/README.md)
+
+- [Input Plugins](./docs/en/guides/How-to-write-input-plugins.md)
+- [Processor Plugins](./docs/en/guides/How-to-write-processor-plugins.md)
+- [Aggregator Plugins](./docs/en/guides/How-to-write-aggregator-plugins.md)
+- [Flusher Plugins](./docs/en/guides/How-to-write-flusher-plugins.md)
+
+
+# Contribution
+
+There are many ways to contribute:
+- [Fix and report bugs](https://github.com/alibaba/ilogtail/issues)
+- [Improve Documentation](https://github.com/alibaba/ilogtail/labels/documentation)
+- [Review code and feature proposals](https://github.com/alibaba/ilogtail/pulls)
+- [Contribute plugins](./docs/en/guides/README.md)
+
+# Contact Us
+You can report bugs, make suggestions or participate in discussions through [Github Issues](https://github.com/alibaba/ilogtail/issues), or contact us with the following ways:
+
+- DingTalk：33867413
+- WeChat：日志服务
+- Bilibli：[阿里云SLS](https://space.bilibili.com/630680534?from=search&seid=2845737427240690794&spm_id_from=333.337.0.0)
+- Zhihu：[阿里云日志服务](https://www.zhihu.com/people/a-li-yun-ri-zhi-fu-wu)
+
+<img src="https://sls-opensource.oss-us-west-1.aliyuncs.com/ilogtail/ilogtail-contact.png?versionId=CAEQMxiBgIC.jPui6RciIDhlNzVjYmQ5OTIwNzQ4NTBiMzk1YTgyN2VkNmU5MDBi" alt="ilogtail logo" style="width: 100%; height: 100%" />
+
+
+
+# Our Users
+Tens of thousands of companies use iLogtail in Alibaba Cloud, IDC, or other clouds. More details please see [here](https://help.aliyun.com/document_detail/250269.html).
+
+# Licence
+[Apache 2.0 License](./LICENSE)