Merge branch 'main' of https://github.com/alibaba/ilogtail into featu…

…re/taiye/add_docker_test

Conflicts:
	pkg/helper/docker_center.go

.github/ISSUE_TEMPLATE/ask-a-question.md

@@ -1,6 +1,6 @@
 ---
 name: Ask a question
-about: Ask anything of ilogtail. It is recommended to ask a question in discussions.
+about: Ask anything of LoongCollector. It is recommended to ask a question in discussions.
 title: "[QUESTION]:"
 labels: question
 assignees: ''
@@ -9,18 +9,18 @@ assignees: ''
 
 Before asking a question, please first consider:
 
-- Searching [existing discussions](https://github.com/alibaba/ilogtail/discussions/)
-- Searching [existing issues](https://github.com/alibaba/ilogtail/issues/)
-- Searching [English Doc](https://github.com/alibaba/ilogtail/tree/main/docs/en)
+- Searching [existing discussions](https://github.com/alibaba/loongcollector/discussions/)
+- Searching [existing issues](https://github.com/alibaba/loongcollector/issues/)
+- Searching [English Doc](https://github.com/alibaba/loongcollector/tree/main/docs/en)
 - Searching Google
 
 -------
 
-提问之前，建议先在 iLogtail 社区寻找答案。
+提问之前，建议先在 LoongCollector 社区寻找答案。
 
-- 查看 [存在的 discussions](https://github.com/alibaba/ilogtail/discussions/)
-- 查看 [存在的 issues](https://github.com/alibaba/ilogtail/issues/)
-- 查看 [中文文档](https://ilogtail.gitbook.io/ilogtail-docs/about/readme)
+- 查看 [存在的 discussions](https://github.com/alibaba/loongcollector/discussions/)
+- 查看 [存在的 issues](https://github.com/alibaba/loongcollector/issues/)
+- 查看 [中文文档](https://github.com/alibaba/loongcollector/tree/main/docs/cn)
 - 查看 Google
 
-如果仍然无法找到答案，我们建议您在[discussions](https://github.com/alibaba/ilogtail/discussions/)进行提问。
+如果仍然无法找到答案，我们建议您在[discussions](https://github.com/alibaba/loongcollector/discussions/)进行提问。

README.md

@@ -1,16 +1,16 @@
-# Alibaba LoongCollector - Fast and Lightweight Observability Data Collector | 中文用户手册（[GitBook](https://ilogtail.gitbook.io/ilogtail-docs/) | [可观测中文社区](https://open.observability.cn/project/ilogtail/about/)）
+# Alibaba LoongCollector - Fast and Lightweight Observability Data Collector | 中文用户手册（[GitBook](https://ilogtail.gitbook.io/ilogtail-docs/) | [可观测中文社区](https://open.observability.cn/project/loongcollector/about/)）
 
 <img src="https://ilogtail-community-edition.oss-cn-shanghai.aliyuncs.com/images/logo/jpg/black-blue.jpg" alt="loongcollector logo" height="100px" align="center" />
 
 LoongCollector was born for observable scenarios and has many production-level features such as lightweight, high performance, and automated configuration, which are widely used internally by Alibaba Group and tens of thousands of external Alibaba Cloud customers. You can deploy it in physical machines, Kubernetes and other environments to collect telemetry data, such as logs, traces and metrics.
 
-[![GitHub contributors](https://img.shields.io/github/contributors/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/contributors)
-[![GitHub stars](https://img.shields.io/github/stars/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/stargazers)
-[![GitHub issues](https://img.shields.io/github/issues/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/issues)
-[![GitHub license](https://img.shields.io/github/license/alibaba/ilogtail)](https://github.com/alibaba/ilogtail/blob/main/LICENSE)
+[![GitHub contributors](https://img.shields.io/github/contributors/alibaba/ilogtail)](https://github.com/alibaba/loongcollector/contributors)
+[![GitHub stars](https://img.shields.io/github/stars/alibaba/ilogtail)](https://github.com/alibaba/loongcollector/stargazers)
+[![GitHub issues](https://img.shields.io/github/issues/alibaba/ilogtail)](https://github.com/alibaba/loongcollector/issues)
+[![GitHub license](https://img.shields.io/github/license/alibaba/ilogtail)](https://github.com/alibaba/loongcollector/blob/main/LICENSE)
 [![Coverity Scan Build Status](https://img.shields.io/coverity/scan/28764.svg)](https://scan.coverity.com/projects/alibaba-ilogtail)
 [![Coverage Status](https://codecov.io/gh/alibaba/ilogtail/branch/main/graph/badge.svg)](https://codecov.io/gh/alibaba/ilogtail)
-[![Go Report Card](https://goreportcard.com/badge/github.com/alibaba/ilogtail)](https://goreportcard.com/report/github.com/alibaba/ilogtail)
+[![Go Report Card](https://goreportcard.com/badge/github.com/alibaba/loongcollector)](https://goreportcard.com/report/github.com/alibaba/loongcollector)
 
 ## Abstract
 
@@ -83,20 +83,20 @@ Our official **User Manual** is located here:
 
 There are many ways to contribute:
 
-* [Fix and report bugs](https://github.com/alibaba/ilogtail/issues)
-* [Improve Documentation](https://github.com/alibaba/ilogtail/labels/documentation)
-* [Review code and feature proposals](https://github.com/alibaba/ilogtail/pulls)
+* [Fix and report bugs](https://github.com/alibaba/loongcollector/issues)
+* [Improve Documentation](https://github.com/alibaba/loongcollector/labels/documentation)
+* [Review code and feature proposals](https://github.com/alibaba/loongcollector/pulls)
 * [Contribute plugins](./docs/en/guides/README.md)
 
 ## Contact Us
 
-You can report bugs, make suggestions or participate in discussions through [Github Issues](https://github.com/alibaba/ilogtail/issues) and [Github Discussions](https://github.com/alibaba/ilogtail/discussions), or contact us with the following ways:
+You can report bugs, make suggestions or participate in discussions through [Github Issues](https://github.com/alibaba/loongcollector/issues) and [Github Discussions](https://github.com/alibaba/loongcollector/discussions), or contact us with the following ways:
 
-* Bilibili：[阿里云SLS](https://space.bilibili.com/630680534?from=search&seid=2845737427240690794&spm_id_from=333.337.0.0)
+* Bilibili：[阿里云SLS](https://space.bilibili.com/630680534)
 * Zhihu：[LoongCollector社区](https://www.zhihu.com/column/c_1533139823409270785)
-* DingTalk：LoongCollector社区
+* DingTalk/WeChat：LoongCollector社区
 
-<img src="https://ilogtail-community-edition.oss-cn-shanghai.aliyuncs.com/images/chatgroup/chatgroup_20240408.png" style="width: 50%; height: 50%" />
+<img src="docs/cn/.gitbook/assets/chatgroup.png" style="width: 60%; height: 60%" />
 
 ## Our Users
 

core/CMakeLists.txt

@@ -114,7 +114,7 @@ include(${CMAKE_CURRENT_SOURCE_DIR}/plugin/flusher/links.cmake)
 
 # Subdirectories (modules). except for common, input, processor, flusher, observer, helper, spl, and provider.
 set(SUB_DIRECTORIES_LIST
-        application app_config checkpoint container_manager metadata logger go_pipeline monitor monitor/metric_constants monitor/profile_sender models
+        application app_config checkpoint container_manager metadata logger go_pipeline monitor monitor/metric_constants monitor/metric_models monitor/profile_sender models
         config config/watcher constants
         pipeline pipeline/batch pipeline/limiter pipeline/plugin pipeline/plugin/creator pipeline/plugin/instance pipeline/plugin/interface pipeline/queue pipeline/route pipeline/serializer
         task_pipeline

core/app_config/AppConfig.cpp

@@ -430,22 +430,6 @@ string GetAgentLogName() {
     }
 }
 
-string GetAgentSnapshotDir() {
-    if (BOOL_FLAG(logtail_mode)) {
-        return GetProcessExecutionDir() + STRING_FLAG(logtail_snapshot_dir);
-    } else {
-        return GetAgentLogDir() + "snapshot";
-    }
-}
-
-string GetAgentStatusLogName() {
-    if (BOOL_FLAG(logtail_mode)) {
-        return "ilogtail_status.LOG";
-    } else {
-        return "loongcollector_status.LOG";
-    }
-}
-
 string GetObserverEbpfHostPath() {
     if (BOOL_FLAG(logtail_mode)) {
         return STRING_FLAG(sls_observer_ebpf_host_path);
@@ -909,7 +893,6 @@ void AppConfig::LoadResourceConf(const Json::Value& confJson) {
         mSendRequestConcurrency = confJson["send_request_concurrency"].asInt();
     else
         mSendRequestConcurrency = INT32_FLAG(send_request_concurrency);
-    LogtailMonitor::GetInstance()->UpdateConstMetric("send_request_concurrency", mSendRequestConcurrency);
 
     if (confJson.isMember("process_thread_count") && confJson["process_thread_count"].isInt())
         mProcessThreadCount = confJson["process_thread_count"].asInt();

core/app_config/AppConfig.h

@@ -48,8 +48,6 @@ std::string GetLocalEventDataFileName();
 std::string GetInotifyWatcherDirsDumpFileName();
 std::string GetAgentLoggersPrefix();
 std::string GetAgentLogName();
-std::string GetAgentSnapshotDir();
-std::string GetAgentStatusLogName();
 std::string GetObserverEbpfHostPath();
 std::string GetSendBufferFileNamePrefix();
 std::string GetLegacyUserLocalConfigFilePath();

core/application/Application.cpp

@@ -70,10 +70,9 @@ DEFINE_FLAG_INT32(tcmalloc_release_memory_interval, "force release memory held b
 DEFINE_FLAG_INT32(exit_flushout_duration, "exit process flushout duration", 20 * 1000);
 DEFINE_FLAG_INT32(queue_check_gc_interval_sec, "30s", 30);
 #if defined(__ENTERPRISE__) && defined(__linux__) && !defined(__ANDROID__)
-DEFINE_FLAG_BOOL(enable_cgroup, "", true);
+DEFINE_FLAG_BOOL(enable_cgroup, "", false);
 #endif
 
-
 DECLARE_FLAG_BOOL(send_prefer_real_ip);
 DECLARE_FLAG_BOOL(global_network_success);
 
@@ -133,7 +132,6 @@ void Application::Init() {
     const string& configIP = AppConfig::GetInstance()->GetConfigIP();
     if (!configIP.empty()) {
         LoongCollectorMonitor::mIpAddr = configIP;
-        LogtailMonitor::GetInstance()->UpdateConstMetric("logtail_ip", GetHostIp());
     } else if (!interface.empty()) {
         LoongCollectorMonitor::mIpAddr = GetHostIp(interface);
         if (LoongCollectorMonitor::mIpAddr.empty()) {
@@ -151,7 +149,6 @@ void Application::Init() {
     const string& configHostName = AppConfig::GetInstance()->GetConfigHostName();
     if (!configHostName.empty()) {
         LoongCollectorMonitor::mHostname = configHostName;
-        LogtailMonitor::GetInstance()->UpdateConstMetric("logtail_hostname", GetHostName());
     }
 
     GenerateInstanceId();
@@ -198,7 +195,6 @@ void Application::Init() {
 
 void Application::Start() { // GCOVR_EXCL_START
     LoongCollectorMonitor::mStartTime = GetTimeStamp(time(NULL), "%Y-%m-%d %H:%M:%S");
-    LogtailMonitor::GetInstance()->UpdateConstMetric("start_time", LoongCollectorMonitor::mStartTime);
 
 #if defined(__ENTERPRISE__) && defined(_MSC_VER)
     InitWindowsSignalObject();

core/common/SafeQueue.h

@@ -74,6 +74,10 @@ class SafeQueue {
         return mQueue.size();
     }
 
+#ifdef APSARA_UNIT_TEST_MAIN
+    void Clear() { std::queue<T>().swap(mQueue); }
+#endif
+
 private:
     std::queue<T> mQueue;
     mutable std::mutex mMux;

core/config/ConfigDiff.h

@@ -38,4 +38,6 @@ using PipelineConfigDiff = ConfigDiff<PipelineConfig>;
 using TaskConfigDiff = ConfigDiff<TaskConfig>;
 using InstanceConfigDiff = ConfigDiff<InstanceConfig>;
 
+enum ConfigDiffEnum { Added, Modified, Removed, Unchanged };
+
 } // namespace logtail

core/config/PipelineConfig.cpp

@@ -191,6 +191,20 @@ bool PipelineConfig::Parse() {
                                mRegion);
         }
         const string pluginType = it->asString();
+        // when input is singleton, there should only one input to simpify config load transaction
+        if (PluginRegistry::GetInstance()->IsGlobalSingletonInputPlugin(pluginType)) {
+            mSingletonInput = pluginType;
+            if (itr->size() > 1) {
+                PARAM_ERROR_RETURN(sLogger,
+                                   alarm,
+                                   "more than 1 input plugin is given when global singleton input plugin is used",
+                                   noModule,
+                                   mName,
+                                   mProject,
+                                   mLogstore,
+                                   mRegion);
+            }
+        }
         if (i == 0) {
             if (PluginRegistry::GetInstance()->IsValidGoPlugin(pluginType)) {
                 mHasGoInput = true;
@@ -241,7 +255,7 @@ bool PipelineConfig::Parse() {
     if (hasFileInput && (*mDetail)["inputs"].size() > 1) {
         PARAM_ERROR_RETURN(sLogger,
                            alarm,
-                           "more than 1 input_file or input_container_stdio plugin is given",
+                           "more than 1 input_file or input_container_stdio is given",
                            noModule,
                            mName,
                            mProject,

core/config/PipelineConfig.h

@@ -32,6 +32,7 @@ struct PipelineConfig {
     uint32_t mCreateTime = 0;
     const Json::Value* mGlobal = nullptr;
     std::vector<const Json::Value*> mInputs;
+    std::optional<std::string> mSingletonInput;
     std::vector<const Json::Value*> mProcessors;
     std::vector<const Json::Value*> mAggregators;
     std::vector<const Json::Value*> mFlushers;
@@ -49,7 +50,8 @@ struct PipelineConfig {
     std::string mLogstore;
     std::string mRegion;
 
-    PipelineConfig(const std::string& name, std::unique_ptr<Json::Value>&& detail) : mName(name), mDetail(std::move(detail)) {}
+    PipelineConfig(const std::string& name, std::unique_ptr<Json::Value>&& detail)
+        : mName(name), mDetail(std::move(detail)) {}
 
     bool Parse();
 
@@ -67,7 +69,8 @@ struct PipelineConfig {
     // bool IsProcessRunnerInvolved() const {
     //     // 长期过渡使用，待C++部分的时序聚合能力与Go持平后恢复下面的正式版
     //     return !(mHasGoInput && !mHasNativeProcessor);
-    //     // return !(mHasGoInput && !mHasNativeProcessor && (mHasGoProcessor || (mHasGoFlusher && !mHasNativeFlusher)));
+    //     // return !(mHasGoInput && !mHasNativeProcessor && (mHasGoProcessor || (mHasGoFlusher &&
+    //     !mHasNativeFlusher)));
     // }
 
     bool HasGoPlugin() const { return mHasGoFlusher || mHasGoProcessor || mHasGoInput; }